Skip to content

Commit dcf1b20

Browse files
celantheclaude
celanthe
and
claude
authored
docs(gitlab): add comprehensive troubleshooting guide (#67)
* docs(gitlab): add comprehensive troubleshooting guide Add troubleshooting.md covering common issues and solutions: - Authentication issues (token expiry, endpoint config) - VM deployment failures (config validation, resource availability) - SSH connection issues (key setup, timeouts, permissions) - Environment variable configuration - Network and connectivity troubleshooting - Job execution problems - Orphaned VM cleanup procedures Also updates README.md to link to the new troubleshooting guide. Addresses DI-342 requirement for troubleshooting documentation. Verified: - All variable names match scripts exactly - Error messages match actual script output - CLI commands verified against Orka3 CLI docs - All reference links validated Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix connectivity test commands in troubleshooting guide Replace non-existent /api/v1/health endpoint references with working CLI-based verification methods (orka3 version). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs(gitlab): address PR review feedback Changes based on ispasov's review: - Remove redundant diagnostics section (Docker image validates these) - Remove orka3 login/user get-token (CI/CD must use service accounts) - Remove token verification steps (not available in container context) - Remove grep piping (use CLI's built-in argument filtering) - Remove export suggestions (vars must be in container/GitLab context) - Consolidate network connectivity to single curl approach - Remove "verify exists" steps (trust CLI error messages) - Add guidance to deploy VMs manually for SSH troubleshooting - Simplify cleanup section (remove stale VM detection complexity) - Remove duplicate ping-based connectivity checks Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: remove incorrect endpoint cause from 401 error section An incorrect endpoint produces timeout errors, not 401s. Endpoint troubleshooting is already covered in the connectivity section. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: deduplicate connectivity check in auth section Reference the network section instead of repeating the curl check. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(gitlab): address PR #67 round 2 review feedback - Add canonical "Setting environment variables" section explaining Option A (GitLab CI/CD Variables) and Option B (Docker container startup), referenced throughout instead of repeating inline instructions - Add EKS note for control planes that don't allow long-lived tokens - Remove -o wide from orka3 node list - Replace VM_DEPLOYMENT_ATTEMPTS yaml snippet with reference to env vars section - Reframe "Invalid ip/port" as a deeper infrastructure issue; remove redundant vm list step (IP/port are in deploy JSON output) - Remove standalone orka3 vm list test-debug step; annotate deploy command with -o json to surface connection details directly - Rename "Screen Sharing (VNC)" to "Screen Sharing or VNC" - Add "SSH key doesn't match" as a cause for Permission denied errors - Add SSH key mounting recommendation in vars section and table row Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
1 parent b1ca4eb commit dcf1b20

2 files changed

Lines changed: 333 additions & 0 deletions

File tree

‎GitLab/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,6 @@
33
The Orka GitLab integration enables you to use [Orka by MacStadium][orka] in your GitLab CI/CI pipelines.
44
Learn how to configure the [GitLab Shell executor](shell-executor.md) or the [GitLab Custom executor](custom-executor.md).
55

6+
For common issues and solutions, see the [Troubleshooting guide](troubleshooting.md).
7+
68
[orka]: https://www.macstadium.com/orka

‎GitLab/troubleshooting.md‎

Lines changed: 331 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,331 @@
1+
# Troubleshooting the GitLab Orka Integration
2+
3+
This guide covers common issues and solutions when using the GitLab [Custom executor][custom] with [Orka][orka].
4+
5+
## Setting environment variables
6+
7+
The integration reads configuration from environment variables. There are two ways to set them:
8+
9+
**Option A — GitLab CI/CD Variables** (available to the build only)
10+
Go to Settings > CI/CD > Variables and add each variable. This is the right place for variables like `ORKA_TOKEN` and `ORKA_CONFIG_NAME` that the runner scripts need during job execution.
11+
12+
**Option B — Docker container startup** (available inside the container)
13+
Pass variables when starting the runner container:
14+
```
15+
docker run --env ORKA_ENDPOINT="http://10.221.188.20" ...
16+
```
17+
Use this for variables that need to be present at the container level, like `ORKA_ENDPOINT`.
18+
19+
For sensitive variables like `ORKA_TOKEN`, enable the **Masked** option in GitLab so the value is hidden in job logs.
20+
21+
For `ORKA_SSH_KEY_FILE`, prefer mounting the key file directly into the runner container (Option B) rather than storing the private key contents as a GitLab variable.
22+
23+
## Authentication issues
24+
25+
### Error: "unauthorized" or "401"
26+
27+
**Symptoms:**
28+
- VM deployment fails with authentication errors
29+
- `orka3` commands return "unauthorized"
30+
31+
**Causes:**
32+
- `ORKA_TOKEN` is invalid or expired
33+
34+
**Solutions:**
35+
36+
1. Generate a new service account token:
37+
```bash
38+
orka3 serviceaccount token <service-account-name>
39+
```
40+
41+
2. Update `ORKA_TOKEN` with the new token. See [Setting environment variables](#setting-environment-variables).
42+
43+
**Note:** Service account tokens are valid for 1 year by default. For custom duration, use `--duration` flag. Some Kubernetes control planes (e.g., EKS) do not allow long-lived tokens. In that case, use the `--no-expiration` flag instead.
44+
45+
### Error: "config not found" or "no such host"
46+
47+
**Symptoms:**
48+
- CLI commands fail before authentication
49+
- "dial tcp: lookup" errors
50+
51+
**Causes:**
52+
- `ORKA_ENDPOINT` is not set or malformed
53+
- Network connectivity issues to Orka API
54+
55+
**Solutions:**
56+
57+
1. Verify the endpoint format (include protocol, no trailing slash). See [Setting environment variables](#setting-environment-variables).
58+
```
59+
# Correct format
60+
http://10.221.188.20
61+
62+
# Incorrect formats
63+
10.221.188.20 # Missing protocol
64+
http://10.221.188.20/ # Trailing slash
65+
```
66+
67+
2. If the endpoint is correct but commands still fail, see [Runner cannot reach Orka endpoint](#runner-cannot-reach-orka-endpoint) for connectivity troubleshooting.
68+
69+
## VM deployment failures
70+
71+
### Error: "VM deployment failed"
72+
73+
**Symptoms:**
74+
- prepare.sh exits with "VM deployment failed"
75+
- Deployment attempts exhausted
76+
77+
**Causes:**
78+
- `ORKA_CONFIG_NAME` doesn't exist or is misspelled
79+
- No available nodes with sufficient resources
80+
81+
**Solutions:**
82+
83+
1. If the error says "config does not exist", check the spelling of `ORKA_CONFIG_NAME` in your GitLab CI/CD Variables. Create the config if needed:
84+
```bash
85+
orka3 vm-config create <config-name> --image <image-name> --cpu <count>
86+
```
87+
88+
2. Check available node resources:
89+
```bash
90+
orka3 node list
91+
```
92+
93+
3. Set `VM_DEPLOYMENT_ATTEMPTS` to your desired retry count. See [Setting environment variables](#setting-environment-variables).
94+
95+
### Error: "Invalid ip" or "Invalid port"
96+
97+
**Symptoms:**
98+
- VM deploys but connection info extraction fails
99+
- "Invalid ip: null" in logs
100+
101+
**Causes:**
102+
- VM deployment returned unexpected JSON format
103+
- VM is in a failed state
104+
105+
**Solutions:**
106+
107+
This usually indicates a deeper infrastructure issue, not something the troubleshooting steps can directly fix.
108+
109+
Deploy a VM manually and inspect the JSON output:
110+
```bash
111+
orka3 vm deploy test-vm --config "$ORKA_CONFIG_NAME" -o json
112+
```
113+
114+
If the manual deploy also returns unexpected output or fails, contact [MacStadium Support][support] with the full error.
115+
116+
Delete the test VM after inspection:
117+
```bash
118+
orka3 vm delete test-vm
119+
```
120+
121+
## SSH connection issues
122+
123+
### Error: "Waited 30 seconds for sshd to start"
124+
125+
**Symptoms:**
126+
- VM deploys successfully
127+
- SSH connection times out after 30 seconds
128+
129+
**Causes:**
130+
- SSH is not enabled on the base image
131+
- SSH key not configured on the VM
132+
- VM is still booting
133+
134+
**Solutions:**
135+
136+
Since the runner automatically deletes failed VMs, deploy a VM manually to troubleshoot:
137+
138+
1. Deploy a test VM:
139+
```bash
140+
# Connection details (IP and SSH port) are in the JSON output
141+
orka3 vm deploy test-debug --config "$ORKA_CONFIG_NAME" -o json
142+
```
143+
144+
2. Connect via Screen Sharing or VNC to check:
145+
- System Preferences > Sharing > Remote Login is enabled
146+
- Your public key is in `~/.ssh/authorized_keys`
147+
148+
3. Test SSH manually:
149+
```bash
150+
ssh -i ~/.ssh/orka_deployment_key -p <PORT> admin@<VM_IP> "echo ok"
151+
```
152+
153+
4. Clean up:
154+
```bash
155+
orka3 vm delete test-debug
156+
```
157+
158+
### Error: "Permission denied (publickey)"
159+
160+
**Symptoms:**
161+
- SSH connection is refused
162+
- "Permission denied" in logs
163+
164+
**Causes:**
165+
- SSH key has a passphrase (not supported)
166+
- Wrong SSH user
167+
- SSH key not in VM's authorized_keys
168+
- SSH key doesn't match the one registered in the VM's authorized_keys
169+
170+
**Solutions:**
171+
172+
1. Verify the SSH key has no passphrase:
173+
```bash
174+
# This should NOT prompt for a passphrase
175+
ssh-keygen -y -f /path/to/key
176+
```
177+
178+
2. If the key has a passphrase, generate a new one without:
179+
```bash
180+
ssh-keygen -t ed25519 -f ~/.ssh/orka_key -N ""
181+
```
182+
183+
3. Verify `ORKA_VM_USER` matches the user on the VM (default: `admin`). See [Setting environment variables](#setting-environment-variables).
184+
185+
4. Deploy a test VM and verify the public key is in `~/.ssh/authorized_keys`.
186+
187+
## Environment variable issues
188+
189+
### Error: "unbound variable" or blank values
190+
191+
**Symptoms:**
192+
- Script fails immediately
193+
- Variables are empty
194+
195+
**Causes:**
196+
- Required environment variables not configured in GitLab
197+
198+
**Solutions:**
199+
200+
See [Setting environment variables](#setting-environment-variables) for how to configure these. Verify all required variables are set:
201+
202+
| Variable | Required | Description |
203+
|----------|----------|-------------|
204+
| `ORKA_TOKEN` | Yes | Service account token |
205+
| `ORKA_ENDPOINT` | Yes | Orka API URL |
206+
| `ORKA_CONFIG_NAME` | Yes | VM config template name |
207+
| `ORKA_SSH_KEY_FILE` | Yes | Private SSH key contents. Recommended: mount the key file into the runner container rather than storing key contents as a GitLab variable. See [Setting environment variables](#setting-environment-variables). |
208+
| `ORKA_VM_USER` | No | SSH user (default: `admin`) |
209+
| `ORKA_VM_NAME_PREFIX` | No | VM name prefix (default: `gl-runner`) |
210+
| `VM_DEPLOYMENT_ATTEMPTS` | No | Retry count (default: `1`) |
211+
212+
For sensitive variables like `ORKA_TOKEN` and `ORKA_SSH_KEY_FILE`, enable the "Masked" option.
213+
214+
## Network and connectivity issues
215+
216+
### Runner cannot reach Orka endpoint
217+
218+
**Symptoms:**
219+
- "Connection refused" or "Connection timed out"
220+
- curl to endpoint fails
221+
222+
**Causes:**
223+
- Runner is not on the same network as Orka
224+
- VPN not connected
225+
- Firewall blocking traffic
226+
227+
**Solutions:**
228+
229+
1. Test connectivity from the runner environment:
230+
```bash
231+
curl -s -o /dev/null -w "%{http_code}" "$ORKA_ENDPOINT/api/v1/cluster-info"
232+
```
233+
234+
2. If using VPN, verify your connection using your [IP plan][ip-plan] details.
235+
236+
3. For Docker-based runners, ensure the container has network access to the Orka endpoint.
237+
238+
### IP mapping issues
239+
240+
**Symptoms:**
241+
- VM deploys but SSH connects to wrong IP
242+
- "No route to host" errors
243+
244+
**Causes:**
245+
- Private/public IP mismatch
246+
- settings.json not configured for IP mapping
247+
248+
**Solutions:**
249+
250+
If your network requires IP mapping, create `/var/custom-executor/settings.json`:
251+
```json
252+
{
253+
"mappings": [
254+
{
255+
"private_host": "10.221.188.100",
256+
"public_host": "203.0.113.100"
257+
}
258+
]
259+
}
260+
```
261+
262+
See [template-settings.md](template-settings.md) for configuration details.
263+
264+
## Job execution issues
265+
266+
### Build script fails
267+
268+
**Symptoms:**
269+
- Job fails during run.sh
270+
- Error is from your CI/CD script, not the integration
271+
272+
**Note:** The integration distinguishes between:
273+
- **Build failures**: Your script failed (returns script exit code)
274+
- **System failures**: Infrastructure failed (returns exit code 1)
275+
276+
If your build script fails, the issue is in your script, not the integration. Test your script on a standalone Orka VM.
277+
278+
### Job hangs or times out
279+
280+
**Symptoms:**
281+
- Job runs but never completes
282+
- GitLab times out the job
283+
284+
**Causes:**
285+
- Long-running process without output
286+
- SSH connection dropped
287+
288+
**Solutions:**
289+
290+
1. For long jobs, add periodic output to prevent GitLab timeout.
291+
292+
2. Consider breaking long jobs into smaller stages.
293+
294+
3. Increase GitLab job timeout in project settings if needed.
295+
296+
## Cleanup issues
297+
298+
### Orphaned VMs
299+
300+
**Symptoms:**
301+
- VMs remain after job completion
302+
303+
**Causes:**
304+
- Runner crashed before cleanup
305+
- Network issue during cleanup
306+
307+
**Solutions:**
308+
309+
Delete orphaned VMs manually:
310+
```bash
311+
orka3 vm delete <vm-name>
312+
```
313+
314+
## Getting help
315+
316+
If you're still experiencing issues:
317+
318+
1. Check the [Orka documentation][orka-docs] for platform-specific guidance
319+
2. Review GitLab Runner [logs][runner-logs]: `gitlab-runner --debug run`
320+
3. Contact [MacStadium Support][support] with:
321+
- Error messages and logs
322+
- Environment details (Runner version, Orka version)
323+
- Steps to reproduce
324+
325+
[custom]: https://docs.gitlab.com/runner/executors/custom.html
326+
[orka]: https://support.macstadium.com/hc/en-us/articles/29904434271387-Orka-Overview
327+
[orka-docs]: https://support.macstadium.com/hc/en-us
328+
[ip-plan]: https://support.macstadium.com/hc/en-us/articles/28230867289883-IP-Plan
329+
[masked-variables]: https://docs.gitlab.com/ee/ci/variables/#mask-a-cicd-variable
330+
[runner-logs]: https://docs.gitlab.com/runner/faq/#how-can-i-get-a-debug-log
331+
[support]: https://support.macstadium.com/

0 commit comments

Comments
 (0)