Skip to content

fix(codeapi-chart): de-hook package-init Job so it runs once, not per-sync #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rschlaefli merged 1 commit into from
Jul 5, 2026
Merged

fix(codeapi-chart): de-hook package-init Job so it runs once, not per-sync #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions helm/codeapi/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,14 +115,17 @@ helm install codeapi . -f values-local.yaml

### 4. Language Packages (Automatic)

The chart includes a **package-init Job** that runs as a Helm `pre-install` hook. It automatically compiles Python, downloads Node/Bun, installs offline package sets, and registers Bash into the packages PVC before the worker pods start.
The chart includes a **package-init Job** that compiles Python, downloads Node/Bun, installs offline package sets, and registers Bash into the packages PVC before the worker pods start. Under Argo CD it is a plain managed resource ordered by `sync-wave: -5` (not a hook), so it runs once and an unchanged sync does not re-run it or wake the (scale-to-zero) sandbox node pool. The completed Job persists (no TTL) and its spec is immutable — forcing a rebuild or bumping runtime versions requires deleting the Job first.

This happens automatically on `helm install`. To force a rebuild:
This happens automatically on `helm install`. To force a rebuild, delete the completed Job first (its spec is immutable, so `helm upgrade`/sync alone cannot patch it), then re-apply with `forceRebuild=true`:

```bash
kubectl delete job codeapi-package-init
helm upgrade codeapi . --set workerSandbox.packages.initJob.forceRebuild=true
```

Under Argo CD, delete the Job the same way and re-sync — it is recreated and re-runs.

To check init job status:

```bash
Expand All @@ -135,6 +138,7 @@ When deploying the `/pkgs` package-root migration, update sandbox env values to
so generated Python/Node/Bun paths are recreated under `/pkgs`:

```bash
kubectl delete job codeapi-package-init
helm upgrade codeapi . --set workerSandbox.packages.initJob.forceRebuild=true
```

Expand Down Expand Up @@ -319,7 +323,8 @@ kubectl describe pod <pod-name>
kubectl get jobs -l app.kubernetes.io/component=package-init
kubectl logs job/codeapi-package-init

# Force a rebuild:
# Force a rebuild (delete the immutable Job first, then re-apply):
kubectl delete job codeapi-package-init
helm upgrade codeapi . --set workerSandbox.packages.initJob.forceRebuild=true

# Then restart sandbox-runner pods
Expand Down
14 changes: 7 additions & 7 deletions helm/codeapi/templates/package-init-job.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,12 @@
{{/*
Package Init Job
Runs before deployments in Argo CD syncs to populate the packages PVC with
Python, Node, Bun, and Bash runtimes for the NsJail sandbox.
Populates the packages PVC (Python, Node, Bun, Bash runtimes for the NsJail
sandbox) once. Ordered before the sandbox-runner by Argo CD sync-wave -5.
Managed resource, NOT a Sync hook: an unchanged sync is an apply no-op that
does NOT re-run the Job or wake the (scale-to-zero) sandbox node pool. No TTL,
so the completed Job persists and stays a no-op across syncs. Its spec is
immutable: to re-run it (forceRebuild or a runtime-version bump), delete the Job
first, then sync (see values.yaml initJob.forceRebuild).
*/}}
{{- if and .Values.workerSandbox.enabled .Values.workerSandbox.packages.initJob.enabled }}
{{- $packageInitNodeSelector := .Values.workerSandbox.sandboxRunner.nodeSelector }}
Expand All @@ -21,14 +26,9 @@ metadata:
{{- include "codeapi.labels" . | nindent 4 }}
app.kubernetes.io/component: package-init
annotations:
"argocd.argoproj.io/hook": Sync
"argocd.argoproj.io/sync-wave": "-5"
"argocd.argoproj.io/hook-delete-policy": BeforeHookCreation,HookSucceeded
spec:
backoffLimit: {{ .Values.workerSandbox.packages.initJob.backoffLimit }}
{{- if .Values.workerSandbox.packages.initJob.ttlSecondsAfterFinished }}
ttlSecondsAfterFinished: {{ .Values.workerSandbox.packages.initJob.ttlSecondsAfterFinished }}
{{- end }}
template:
metadata:
labels:
Expand Down
11 changes: 8 additions & 3 deletions helm/codeapi/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -293,10 +293,16 @@ workerSandbox:
# If you have pre-built packages, specify the existing claim
existingClaim: ""

# Init job to populate packages before sandbox deployments in Argo CD syncs
# Compiles Python from source and downloads Node/Bun binaries
# Init job to populate packages before sandbox deployments in Argo CD syncs.
# Compiles Python from source and downloads Node/Bun binaries. Managed
# resource ordered by sync-wave, not a Sync hook — see package-init-job.yaml
# for why. No ttlSecondsAfterFinished: a TTL-deleted Job would be recreated
# on the next sync and wake the sandbox pool.
initJob:
enabled: true
# The Job spec is immutable and (de-hooked) persists after completion. To
# force a rebuild or change runtime versions, delete the existing Job first
# (kubectl delete job <release>-package-init), then set true / bump and sync.
forceRebuild: false
image:
repository: codeapi-package-init
Expand All @@ -307,7 +313,6 @@ workerSandbox:
bunVersion: "1.3.14"
bashPackageVersion: "5.2.0"
backoffLimit: 2
ttlSecondsAfterFinished: 3600
resources:
requests:
cpu: 500m
Expand Down