Skip to content

Commit 8b1de15

Browse files
sklinglernvsklinglernv
authored
Merge pull request #163 from NVIDIA-NeMo/nooa-release-process
Add CI-controlled release gate
2 parents c2bbdc7 + b8a5e7a commit 8b1de15

3 files changed

Lines changed: 1612 additions & 200 deletions

File tree

‎RELEASING.md‎

Lines changed: 119 additions & 67 deletions
Original file line numberDiff line numberDiff line change
@@ -1,76 +1,128 @@
11
# Releasing
22

3-
`nooa`, `nooa-cli`, `nooa-memory` and `nooa-bench` release together from the
4-
same commit. **The version comes from the git tag** — there is no `version =`
5-
in any `pyproject.toml` and no bump step. On tag `v0.0.9` the wheels are
6-
`0.0.9`; between tags they are `0.0.9.devN`.
7-
8-
## Cutting a release
9-
10-
```bash
11-
git checkout main && git pull
12-
uv run python scripts/make_release.py v0.0.9
13-
```
14-
15-
The script runs everything in order and stops at two prompts — after the
16-
capability report, and after the draft notes:
17-
18-
| Step | Fails the run? |
19-
|---|---|
20-
| Preflight — on `main`, clean, in sync with origin, tag unused | yes |
21-
| Lint, SPDX headers, unit tests | yes |
22-
| Build 4 wheels, version == tag, smoke import | yes |
23-
| Capability diff vs the previous release, 4 models × 3 runs | only below the floor |
24-
| `gh release create --draft`, capability report appended | — |
25-
| `gh release edit --draft=false` → triggers `publish.yml` | — |
26-
27-
Publishing uploads to PyPI via Trusted Publishing. Each package waits on its
28-
`pypi-<package>` GitHub Environment, so a reviewer approves before upload.
29-
30-
### Capability gate
31-
32-
Both arms run fresh: HEAD in the working tree, the previous tag in a temporary
33-
worktree. Comparing against a stored baseline cannot tell a real regression
34-
from the endpoint behind a model alias changing.
35-
36-
The only hard threshold is a **floor** on the stable tier (60%) — clearing it
37-
requires typing `OVERRIDE`. Everything else (collapses, new error types, drops
38-
beyond ±5 points) is reported for a human to judge. An arm where >50% of
39-
samples error is rejected as infrastructure failure rather than reported as a
40-
result. Results cache under `tmp/release-check/`, so aborting at a prompt does
41-
not mean paying for another run.
42-
43-
### Flags
44-
45-
| Flag | Use |
46-
|---|---|
47-
| `--checks-only` | run everything, print the report, touch nothing |
48-
| `--dry-run` | print the `gh` commands instead of running them |
49-
| `--skip-capability` | docs-only releases |
50-
| `--models` / `--runs` / `--limit` | cheap rehearsal; requires `--checks-only` |
51-
52-
Rehearse without spending real money:
3+
`nooa`, `nooa-cli`, `nooa-memory`, and `nooa-bench` release together from one
4+
commit. The version comes from the Git tag: on `v0.0.10` the distributions are
5+
`0.0.10`; between tags they are development versions.
6+
7+
## Normal release path
8+
9+
Releases are gated by a manually started pipeline on protected `main` in the
10+
private `interactive-agents/nooa-dev` GitLab project. Supply both:
11+
12+
- `NOOA_RELEASE_TAG`: a new canonical `vX.Y.Z` version;
13+
- `NOOA_RELEASE_SHA`: the full 40-character SHA of a commit currently reachable
14+
from public GitHub `main`.
15+
16+
The controller freezes that SHA. A later push to `main` does not change the
17+
candidate under test. The job builds the internal NVIDIA model-alias wheel from
18+
the controller commit, clones this repository at the candidate, and calls
19+
`scripts/make_release.py --ci`. The public runner remains the implementation of
20+
the gate; GitLab YAML only provisions and invokes it.
21+
22+
The strict gate performs:
23+
24+
1. Ruff lint and formatting, SPDX checks, unit tests, and explicit OS sandbox
25+
containment tests.
26+
2. Builds all four wheels and source distributions under a temporary local tag,
27+
verifies their versions, and smoke-tests imports and `nooa --version` in a
28+
clean environment.
29+
3. Runs the full capability suite for the candidate and previous release, fresh
30+
and back-to-back: four gate models, three runs, full data, no response cache.
31+
4. Writes private results, traces, distributions, checksums, a JSON manifest,
32+
and sanitized public notes to the GitLab job artifacts.
33+
5. After every hard gate passes, creates or safely updates one GitHub **draft**
34+
targeting the exact tested SHA.
35+
36+
The candidate and baseline environments receive the same explicit
37+
`nemo-oo-agents-nvidia` wheel. Strict CI never copies `.env`, discovers ambient
38+
packages, or reuses capability results from another candidate. Each sample,
39+
stalled run, and complete arm has a timeout; completed JSONL evidence remains
40+
available when a later sample fails.
41+
42+
The stable-tier 60% floor, unusable/missing results, and an arm with more than
43+
50% errors are hard failures. Collapses, new errors, removed tests, and drops
44+
beyond the noise band are advisory: they are prominent in the draft but remain
45+
a human judgment when hard gates pass.
46+
47+
## Human approval
48+
49+
The GitLab job can create only a draft. It has no command that publishes a
50+
release or uploads to PyPI.
51+
52+
- **Accept:** inspect the draft and private GitLab evidence, then click GitHub's
53+
**Publish release** button.
54+
- **Reject:** delete the draft with its tag, or leave it unpublished until a
55+
maintainer performs cleanup. A subsequent pipeline may reuse only a draft
56+
whose target is the identical frozen SHA; any different target fails closed.
57+
58+
Publishing is the single human approval. `.github/workflows/publish.yml` listens
59+
for `release: published` and automatically rebuilds, smoke-tests, and uploads
60+
all four packages to PyPI using Trusted Publishing. Despite their names, the
61+
current `pypi-*` GitHub Environments have no configured reviewer protection, so
62+
there is no second approval after **Publish release**.
63+
64+
## Evidence and recovery
65+
66+
GitLab retains the release evidence for 90 days. Start with `job-summary.md` and
67+
`release-manifest.json`; the latter indexes exact commits, lock/config hashes,
68+
toolchain identity, check outcomes, capability scope/results, distribution
69+
checksums, and draft identity. Raw `.noo-eval.jsonl` files and traces are private
70+
artifacts and must not be copied into the public draft.
71+
72+
Hard-gate failure or capability-infrastructure failure before draft creation
73+
creates no draft. Fix the problem and start a new pipeline with the same tag
74+
and SHA. A failure after `gh release create` or `gh release edit` may leave a
75+
draft requiring reconciliation; rerun to update an exact matching draft without
76+
duplication. If a draft has the wrong target or a release is already published,
77+
the runner stops instead of mutating it.
78+
79+
Use the controller's one-model/one-run/one-sample rehearsal mode to exercise
80+
setup and reporting cheaply; rehearsals can never create a draft. Before first
81+
production use, run the complete gate once with draft creation disabled and
82+
review all artifacts.
83+
84+
Before requesting review of coordinated release-process changes, the private
85+
controller may run an unmerged rehearsal against a canonical GitHub
86+
`refs/pull/<number>/head` ref and its exact SHA. That mode may relax only the
87+
candidate's reachability from GitHub `main`; it remains reduced-scope, requires
88+
the private controller to authenticate the pull ref, runs every deterministic,
89+
containment, build, smoke, and evidence check, and can never create a draft.
90+
91+
## Publication rehearsal and artifact promotion
92+
93+
A manual run of `publish.yml` always targets TestPyPI and is the safe way to
94+
rehearse its build, OIDC, and upload jobs without consuming a production PyPI
95+
version. The production trigger remains `release: published`; do not publish a
96+
throwaway GitHub release to test it because that event intentionally reaches
97+
real PyPI.
98+
99+
The current first increment rebuilds after publication from the exact published
100+
tag. The candidate artifacts and checksums are retained for review, but they are
101+
not yet promoted. A follow-up should attach the checked artifacts to the draft
102+
and make `publish.yml` download and verify those exact files while preserving
103+
the existing PyPI OIDC identities.
104+
105+
## Emergency local fallback
106+
107+
If the private controller is unavailable, a trusted maintainer may run the
108+
public script from a clean, current `main` checkout:
53109

54110
```bash
55-
uv run python scripts/make_release.py v0.0.9 --checks-only \
56-
--models claude-haiku --runs 1 --limit 1
111+
uv run python scripts/make_release.py v0.0.10
57112
```
58113

59-
Report logic is covered by `tests/test_make_release.py`.
60-
61-
## One-time PyPI setup
114+
This is deliberately narrower than the old workflow: it may create a draft but
115+
cannot publish it. It retains local compatibility for model aliases, prompts
116+
before drafting advisory results, and still blocks the capability floor. Use it
117+
only for recovery, record the evidence separately, and publish only through the
118+
GitHub draft UI.
62119

63-
Each project needs a pending publisher at
64-
<https://pypi.org/manage/account/publishing/> — owner `NVIDIA-NeMo`, repo
65-
`labs-OO-Agents`, workflow `publish.yml`, and a **distinct environment per
66-
package**: `pypi-nooa`, `pypi-nooa-cli`, `pypi-nooa-memory`, `pypi-nooa-bench`.
67-
PyPI keys a pending publisher on (owner, repo, workflow, environment), so a
68-
shared environment makes the second registration fail. The matching GitHub
69-
Environments must exist too.
120+
## Trusted Publishing setup
70121

71-
Repeat on <https://test.pypi.org> with `testpypi-<package>` names. Running the
72-
**Publish** workflow manually always targets TestPyPI.
122+
Each project needs a publisher configured for owner `NVIDIA-NeMo`, repository
123+
`labs-OO-Agents`, workflow `publish.yml`, and its distinct environment:
124+
`pypi-nooa`, `pypi-nooa-cli`, `pypi-nooa-memory`, or `pypi-nooa-bench`. Repeat
125+
with `testpypi-*` environments on TestPyPI.
73126

74-
> Every `uses:` in `publish.yml` must be an `actions/*` action. This org
75-
> enforces an allowlist, and a disallowed action fails the whole workflow at
76-
> startup — that is what left CI dead for eight days (PR #50).
127+
Every `uses:` entry in `publish.yml` must remain compatible with the NVIDIA
128+
organization's GitHub Actions allowlist.

0 commit comments

Comments
 (0)