Impact of the new feature
Following the CI/CD compromise, GitHub Actions is currently disabled on this repository. This issue tracks the controls we want in place before we turn it back on, plus the Harbor registry migration that has to land in the same window.
Beyond the original incident, the release workflow contains a live remote-code-execution path:
# .github/workflows/docker_images_template.yaml
curl -ksLO https://raw.githubusercontent.com/dmwm/WMCore/master/bin/docker.sh
chmod +x docker.sh
./docker.sh build ... # executed in a job holding the CERN registry robot token
The job downloads a script from master at runtime and executes it while authenticated to the registry. Anyone who can land a commit on master — which is exactly what happened in the incident — gets arbitrary code execution with the Harbor credential in scope. Gating who can trigger releases does not fix this; a perfectly legitimate release tag would still execute an attacker's docker.sh.
The same job also runs ankitvgupta/ref-to-tag-action@master — an unpinned third-party action on a mutable branch of a personal repository, in that same credential-bearing job.
Is your feature request related to a problem? Please describe.
Incident report https://gist.github.com/vkuznet/c070afa3f68c8ccdc88ee45ec0f4d100
Describe the solution you'd like
Proposed changes
Code
Repository / org settings
Additional context
What this changes for day to day opeations
- Opening a PR / getting it reviewed: unchanged, except master now requires one approving review — no more direct pushes, including for admins.
- Cutting a release: unchanged up to the tag push, but the tag must be pushed by someone on the release team, and the pipeline then pauses for a human approval before it can touch PyPI or the registry. One click releases all 11 image jobs.
- CI on PRs: unchanged for maintainers. First-time external contributors' PRs need a maintainer to click "Approve and run".
Scenarios
A. External contributor opens a PR from their fork
Their PR appears normally. CI does not start; the PR shows "1 workflow awaiting approval" and a maintainer must click Approve and run. Rationale: a fork PR can modify workflow files, and you don't want a stranger's YAML executing on your runners before a human has read it. Once approved, CI runs — but with a read-only GITHUB_TOKEN and no access to CERN_LOGIN/CERN_TOKEN (they're environment secrets now, and PR runs aren't on a ..* ref so they can't request the environment). Approval is per-PR for first-time contributors; once they've had a PR merged, GitHub stops asking.
B. Maintainer opens a PR from a branch in dmwm/WMCore
CI starts immediately, no approval prompt (they're not an external contributor). Unit tests, pylint, whatever runs today runs the same. The only new friction: master now needs one approving review to merge. You can no longer push straight to master, and neither can an admin — the bypass list is empty on purpose. This is the single most visible day-to-day change.
C. Merging a PR to master
Nothing publishes. No tag, no trigger. pypi_build_and_images.yaml fires only on push: tags: and workflow_dispatch, so a merge to master builds nothing and pushes nothing to PyPI or Harbor. Same as today.
D. Release manager cuts 2.4.6
./bin/buildrelease.sh ... # → git push origin 2.4.6
- Tag ruleset checks the pusher against the bypass list → they're on the wmcore-release team → allowed.
- Tag push fires pypi_build_and_images.yaml.
- build_and_publish_services (13 matrix jobs, environment: production) → all pause. The run shows a yellow ⏸ "Review pending deployments" banner.
- You (or your co-reviewer — you can approve your own, though a second pair of eyes is the point) click Review deployments → production → Approve and deploy. One click releases all 13 matrix jobs, not one click each.
- Packages build and publish to PyPI via OIDC.
- build_images (11 matrix jobs, environment: registry) → pauses again, this time for registry.
- Approve → the robot credential is injected → each job checks out WMCore at tag 2.4.6, clones the Dockerfile from CMSKubernetes, sleeps 5min for PyPI propagation, builds, and pushes to registry.cern.ch/cmswmcore/<servic
- Because 2.4.6 matches X.Y.Z, does 2.4.6-stable.
Net new friction: two clicks per release.
E. Someone not on the release team
! [remote rejected] 2.4.6 -> 2.4.6 (refusing to allow tag creation - Restrict creations)
Rejected at the git layer. No workflow, no run, nothing to review. Note this is the case branch protection would not have caught — branch rules don't apply to tags at all.
F. Manually starting the pipeline (workflow_dispatch)
Actions tab → Build and Upload to PyPI → Run workflow → the ref dropdown. Pick a tag, not a branch.
- Tag selected (e.g. 2.4.6): works exactly as scenario D — same two approval gates. This is your re-run path when a release half-fails (e.g. the classic sleep 300 PyPI-propagation race — the packages published but some image jobs 404'd on pip install). Re-dispatch on the same tag and only re-approve.
- Branch selected (e.g. master): the job now fails immediately with Refusing to build: refs/heads/master is not a tag ref. Previously this produced an empty PYPI_TAG and built garbage images. Failing loudly is the improvement.
Note the tag ruleset does not gate workflow_dispatch — anyone with write access can dispatch. The environment reviewers are what stop them. That's why the environment gate is the backbone and the tag ruleset is the perimeter.
G. The incident, replayed against these controls
An attacker with a compromised maintainer PAT:
┌─────────────────────────────────────────┬───────────────────────────────────────────────────┐
│ Attempt │ Result │
├─────────────────────────────────────────┼───────────────────────────────────────────────────┤
│ Push malicious commit to master │ Blocked — branch ruleset, no admin bypass │
├─────────────────────────────────────────┼───────────────────────────────────────────────────┤
│ Push commit to a side branch + tag it │ Blocked — tag ruleset, not on release team │
│ 2.4.6 │ │
├─────────────────────────────────────────┼───────────────────────────────────────────────────┤
│ Open a PR with a malicious workflow │ Runs CI, but read-only token, no secrets │
│ │ reachable │
├─────────────────────────────────────────┼───────────────────────────────────────────────────┤
│ workflow_dispatch the release pipeline │ Runs, then stalls at the approval gate — never │
│ │ sees the robot token │
├─────────────────────────────────────────┼───────────────────────────────────────────────────┤
│ Land code on master to poison the │ Path deleted — script now comes from the │
│ curl'd docker.sh │ signed-off tag │
└─────────────────────────────────────────┴───────────────────────────────────────────────────┘
Every path either dead-ends or lands in front of a human.
Impact of the new feature
Following the CI/CD compromise, GitHub Actions is currently disabled on this repository. This issue tracks the controls we want in place before we turn it back on, plus the Harbor registry migration that has to land in the same window.
Beyond the original incident, the release workflow contains a live remote-code-execution path:
The job downloads a script from master at runtime and executes it while authenticated to the registry. Anyone who can land a commit on master — which is exactly what happened in the incident — gets arbitrary code execution with the Harbor credential in scope. Gating who can trigger releases does not fix this; a perfectly legitimate release tag would still execute an attacker's docker.sh.
The same job also runs ankitvgupta/ref-to-tag-action@master — an unpinned third-party action on a mutable branch of a personal repository, in that same credential-bearing job.
Is your feature request related to a problem? Please describe.
Incident report https://gist.github.com/vkuznet/c070afa3f68c8ccdc88ee45ec0f4d100
Describe the solution you'd like
Proposed changes
Code
Repository / org settings
Additional context
What this changes for day to day opeations
Scenarios
A. External contributor opens a PR from their fork
Their PR appears normally. CI does not start; the PR shows "1 workflow awaiting approval" and a maintainer must click Approve and run. Rationale: a fork PR can modify workflow files, and you don't want a stranger's YAML executing on your runners before a human has read it. Once approved, CI runs — but with a read-only GITHUB_TOKEN and no access to CERN_LOGIN/CERN_TOKEN (they're environment secrets now, and PR runs aren't on a ..* ref so they can't request the environment). Approval is per-PR for first-time contributors; once they've had a PR merged, GitHub stops asking.
B. Maintainer opens a PR from a branch in dmwm/WMCore
CI starts immediately, no approval prompt (they're not an external contributor). Unit tests, pylint, whatever runs today runs the same. The only new friction: master now needs one approving review to merge. You can no longer push straight to master, and neither can an admin — the bypass list is empty on purpose. This is the single most visible day-to-day change.
C. Merging a PR to master
Nothing publishes. No tag, no trigger. pypi_build_and_images.yaml fires only on push: tags: and workflow_dispatch, so a merge to master builds nothing and pushes nothing to PyPI or Harbor. Same as today.
D. Release manager cuts 2.4.6
./bin/buildrelease.sh ... # → git push origin 2.4.6
Net new friction: two clicks per release.
E. Someone not on the release team
! [remote rejected] 2.4.6 -> 2.4.6 (refusing to allow tag creation - Restrict creations)
Rejected at the git layer. No workflow, no run, nothing to review. Note this is the case branch protection would not have caught — branch rules don't apply to tags at all.
F. Manually starting the pipeline (workflow_dispatch)
Actions tab → Build and Upload to PyPI → Run workflow → the ref dropdown. Pick a tag, not a branch.
Note the tag ruleset does not gate workflow_dispatch — anyone with write access can dispatch. The environment reviewers are what stop them. That's why the environment gate is the backbone and the tag ruleset is the perimeter.
G. The incident, replayed against these controls
An attacker with a compromised maintainer PAT:
Every path either dead-ends or lands in front of a human.