docs: codify the release confidence process with executable tooling

[lfnovo]

Summary

Preserves the release process designed and executed for v1.11.0 as versioned, reproducible muscle:

Docs

• .github/RELEASE_PROCESS.md v2 — adds the confidence process to the mechanics: changelog audit → risk-based test matrix (buckets A automated / B automatable-with-investment / C release-owner) → image gate (fresh install + upgrade against real containers) → fix loop with an explicit re-test policy → CI-based publishing → communication structure with a mandatory credits section → retro. Plus the gotchas that cost real iterations this cycle (API_URL crossover, host.docker.internal, SurrealDB import quirks).
• ADR-005 — records the decision and the empirical evidence: the v1.11.0 unit suite was fully green while sort_by=title 500'd (SEARCH-index interaction) and credential clearing silently no-oped (mirror frontend/API bugs) — both caught only by this process.

Tooling (scripts/release-test/ + make targets)

• make release-test TAG= OLD_TAG= — automated image gate: fresh install (migrations, in-image worker, nginx-proxied checks) and upgrade-with-data scenarios, with per-phase ports and container-identity assertions
• make release-stack TAG= [DUMP=] — browsable RC stack on the local machine, optionally importing a copy of dev data, for the manual pushed-image verification
• make release-stack-down — teardown

Testing

• bash -n on both scripts; make target arg-guards verified
• The harness itself ran 28/28 green twice during v1.11.0 (against :local and against the final 1.11.0 image)
• scripts/check_md_links.py: all relative links resolve

[cubic-dev-ai]

7 issues found across 8 files

Confidence score: 2/5

• The highest risk is false-green release validation across scripts/release-test/docker-compose.release-test.yml, scripts/release-test/rc-stack.sh, and scripts/release-test/release-image-test.sh: UI checks can hit the wrong API port, unready stacks can still report success, and invalid modes can exit cleanly, so broken images may be approved. Wire each phase to an explicit RC_API_URL, fail hard after readiness retries, and add rejecting default branches before merging.
• In scripts/release-test/docker-compose.release-test.yml, using floating v2 for SurrealDB makes release outcomes non-reproducible, so the same commit can pass or fail depending on when CI runs. Pin the DB image version (ideally digest) to stabilize the gate before merging.
• scripts/release-test/nginx.conf lacks client_max_body_size, so uploads over 1MB return 413 and the documented “big upload” verification can fail for proxy reasons rather than image quality. Set an explicit upload limit aligned with the test scenario before merging.
• Makefile forwarding of DUMP and action handling in scripts/release-test/rc-stack.sh can silently no-op or mis-handle paths with spaces/globs, causing confusing import behavior and missed stack operations. Quote DUMP and make unknown actions exit nonzero to reduce operator error before merge.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="scripts/release-test/docker-compose.release-test.yml">

<violation number="1" location="scripts/release-test/docker-compose.release-test.yml:3">
P2: Release-gate results can change without a repository change because `v2` floats to later database releases. Pin the tested SurrealDB version, preferably to an image digest, so a release run is reproducible.</violation>

<violation number="2" location="scripts/release-test/docker-compose.release-test.yml:18">
P1: Browser UI checks against this release stack resolve their API to port 5055, not the stack's API port (15055/25055); they can hit a local dev API or fail. Pass each phase's API URL as `RC_API_URL` (or make it required) and assert the returned `/config` value.</violation>
</file>

<file name="scripts/release-test/nginx.conf">

<violation number="1" location="scripts/release-test/nginx.conf:5">
P2: Uploads >1MB through the nginx proxy will silently return a 413 error because `client_max_body_size` is not set (defaults to 1MB). The release process specifically calls out "big upload" as a scenario to verify, so this default could block that part of testing. Consider adding `client_max_body_size` with a reasonable limit (e.g. 100M) to avoid confusing failures during release validation.</violation>
</file>

<file name="Makefile">

<violation number="1" location="Makefile:61">
P3: Dump paths containing spaces or shell glob characters are split before `rc-stack.sh` receives them, causing import to fail or select an unintended file. Quote `DUMP` when forwarding it.</violation>
</file>

<file name="scripts/release-test/rc-stack.sh">

<violation number="1" location="scripts/release-test/rc-stack.sh:36">
P3: A misspelled action such as `rc-stack.sh uup <tag>` silently succeeds without creating or removing a stack. Add a default case that prints usage and exits nonzero.</violation>

<violation number="2" location="scripts/release-test/rc-stack.sh:61">
P2: An app that never becomes ready is reported as a usable RC stack and this command exits successfully. Fail after the final retry so release verification cannot proceed against a broken image.</violation>
</file>

<file name="scripts/release-test/release-image-test.sh">

<violation number="1" location="scripts/release-test/release-image-test.sh:146">
P2: A misspelled mode reports `0 passed, 0 failed` and exits successfully, so CI can skip the image gate entirely. Add a rejecting default branch.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: Browser UI checks against this release stack resolve their API to port 5055, not the stack's API port (15055/25055); they can hit a local dev API or fail. Pass each phase's API URL as RC_API_URL (or make it required) and assert the returned /config value.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/release-test/docker-compose.release-test.yml, line 18:

<comment>Browser UI checks against this release stack resolve their API to port 5055, not the stack's API port (15055/25055); they can hit a local dev API or fail. Pass each phase's API URL as `RC_API_URL` (or make it required) and assert the returned `/config` value.</comment>

<file context>
@@ -0,0 +1,43 @@
+      # Without an explicit API_URL the frontend's /config points the BROWSER
+      # at host:5055 — on a dev machine that is the development API, not this
+      # stack (silent data crossover between environments!)
+      - API_URL=${RC_API_URL:-}
+      - OPEN_NOTEBOOK_ENCRYPTION_KEY=${RC_ENCRYPTION_KEY:-release-test-key}
+      - SURREAL_URL=ws://surrealdb:8000/rpc
</file context>

[cubic-dev-ai]

P2: Release-gate results can change without a repository change because v2 floats to later database releases. Pin the tested SurrealDB version, preferably to an image digest, so a release run is reproducible.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/release-test/docker-compose.release-test.yml, line 3:

<comment>Release-gate results can change without a repository change because `v2` floats to later database releases. Pin the tested SurrealDB version, preferably to an image digest, so a release run is reproducible.</comment>

<file context>
@@ -0,0 +1,43 @@
+services:
+  surrealdb:
+    image: surrealdb/surrealdb:v2
+    command: start --log info --user root --pass root rocksdb:/mydata/test.db
+    user: root
</file context>

[cubic-dev-ai]

P2: Uploads >1MB through the nginx proxy will silently return a 413 error because client_max_body_size is not set (defaults to 1MB). The release process specifically calls out "big upload" as a scenario to verify, so this default could block that part of testing. Consider adding client_max_body_size with a reasonable limit (e.g. 100M) to avoid confusing failures during release validation.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/release-test/nginx.conf, line 5:

<comment>Uploads >1MB through the nginx proxy will silently return a 413 error because `client_max_body_size` is not set (defaults to 1MB). The release process specifically calls out "big upload" as a scenario to verify, so this default could block that part of testing. Consider adding `client_max_body_size` with a reasonable limit (e.g. 100M) to avoid confusing failures during release validation.</comment>

<file context>
@@ -0,0 +1,18 @@
+# streaming fix must work anyway because the API sends X-Accel-Buffering: no,
+# which nginx honors. Host is passed through verbatim to exercise the
+# frontend's Host validation with a real proxy in the path.
+server {
+    listen 80;
+    server_name _;
</file context>

[cubic-dev-ai]

P2: An app that never becomes ready is reported as a usable RC stack and this command exits successfully. Fail after the final retry so release verification cannot proceed against a broken image.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/release-test/rc-stack.sh, line 61:

<comment>An app that never becomes ready is reported as a usable RC stack and this command exits successfully. Fail after the final retry so release verification cannot proceed against a broken image.</comment>

<file context>
@@ -0,0 +1,75 @@
+    fi
+    compose up -d
+    echo "Waiting for API..."
+    for i in $(seq 1 30); do
+      curl -sf -m 5 -o /dev/null http://localhost:15055/docs && break
+      sleep 5
</file context>

[cubic-dev-ai]

P2: A misspelled mode reports 0 passed, 0 failed and exits successfully, so CI can skip the image gate entirely. Add a rejecting default branch.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/release-test/release-image-test.sh, line 146:

<comment>A misspelled mode reports `0 passed, 0 failed` and exits successfully, so CI can skip the image gate entirely. Add a rejecting default branch.</comment>

<file context>
@@ -0,0 +1,150 @@
+case "${1:-all}" in
+  fresh)   fresh_test ;;
+  upgrade) upgrade_test ;;
+  all)     fresh_test; upgrade_test ;;
+esac
+
</file context>

[cubic-dev-ai]

P3: Dump paths containing spaces or shell glob characters are split before rc-stack.sh receives them, causing import to fail or select an unintended file. Quote DUMP when forwarding it.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At Makefile, line 61:

<comment>Dump paths containing spaces or shell glob characters are split before `rc-stack.sh` receives them, causing import to fail or select an unintended file. Quote `DUMP` when forwarding it.</comment>

<file context>
@@ -43,6 +44,25 @@ docker-buildx-clean:
+# Usage: make release-stack TAG=1.12.0 [DUMP=/tmp/dev-dump.surql]
+release-stack:
+	@test -n "$(TAG)" || (echo "usage: make release-stack TAG=<tag> [DUMP=<dump.surql>]"; exit 1)
+	bash scripts/release-test/rc-stack.sh up "$(TAG)" $(DUMP)
+
+release-stack-down:
</file context>

[cubic-dev-ai]

P3: A misspelled action such as rc-stack.sh uup <tag> silently succeeds without creating or removing a stack. Add a default case that prints usage and exits nonzero.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/release-test/rc-stack.sh, line 36:

<comment>A misspelled action such as `rc-stack.sh uup <tag>` silently succeeds without creating or removing a stack. Add a default case that prints usage and exits nonzero.</comment>

<file context>
@@ -0,0 +1,75 @@
+  docker compose -p onrelrc -f "$DIR/docker-compose.release-test.yml" "$@"
+}
+
+case "$1" in
+  down)
+    compose down -v
</file context>