docs: codify the release confidence process with executable tooling

.github/RELEASE_PROCESS.md

@@ -4,15 +4,18 @@ Open Notebook uses a flow-driven release process. Work moves from `ready`
 issues into pull requests, pull requests merge to `main`, and maintainers cut a
 version when the branch has enough validated change to ship.
 
+This document covers both the **mechanics** (how to cut, build and publish) and
+the **confidence process** (how we know a release is good before users get it).
+It was redesigned during the v1.11.0 release ([ADR-005](../docs/7-DEVELOPMENT/decisions/ADR-005-release-confidence-process.md)).
+
 ## Release Model
 
 - Patch releases ship backwards-compatible fixes.
 - Minor releases ship backwards-compatible features and improvements.
 - Major releases are planned with a milestone when they include breaking
   changes or migrations that need user coordination.
-- Release candidates and community soak labels are no longer part of the
-  process. Use the `in-dev-build` label for changes available in development
-  images and `released` for shipped work.
+- Use the `in-dev-build` label for changes available in development images and
+  `released` for shipped work.
 
 ## Normal Flow
 
@@ -21,59 +24,132 @@ version when the branch has enough validated change to ship.
 3. Merge the pull request after review and required checks pass.
 4. Let the development build publish the `v1-dev` image from `main`.
 5. Cut a stable release when `main` has a coherent set of changes ready for
-   users.
+   users — following the confidence process below.
 
-## Cutting A Stable Release
+## The Confidence Process
+
+Releases keep getting bigger; ad-hoc verification does not scale. Before
+cutting, run this sequence:
+
+### 0. Changelog audit
+
+Diff `git log <last-tag>..main` against the `[Unreleased]` section of the
+CHANGELOG. Every merged PR must be represented (entries reference the issue
+number when one exists, the PR number otherwise). The changelog is the input
+for both the test plan and the release notes — close the gaps first, via PR.
 
-1. Confirm `main` is green and review the changes since the previous release.
-2. Update `pyproject.toml` with the target semantic version.
-3. Move the relevant `CHANGELOG.md` entries from `Unreleased` into a dated
-   version section.
-4. Create a GitHub release for the version tag, for example `v1.11.0`.
-5. Run the `Build and Release` workflow. Push `v1-latest` tags for normal
-   stable releases.
-6. Verify the published images are available in GHCR and Docker Hub when
-   Docker Hub credentials are configured.
-7. Mark shipped issues with `released` and close any release-tracking tasks.
+### 1. Risk-based test matrix
 
-## Docker Image Publishing
+Build a matrix from the actual release diff: each change → what it can break
+and for whom → which bucket tests it. Pay special attention to
+**"does the protection break legitimate use?"** for security changes (e.g. an
+SSRF guard vs. self-hosted Ollama on localhost) and to anything a reverse
+proxy, an upgrade, or a big upload would exercise.
 
-The publishing workflow, from local test to promoted release:
+Buckets:
+
+- **A — automated, high confidence, run now**: full backend suite, frontend
+  lint/tests/production build, the smoke-e2e agent (full API happy path + UI
+  verification), targeted regression probes for the release's specific risks,
+  dependency audit.
+- **B — automatable with investment**: decide per item whether to build the
+  muscle now (it compounds: the image gate below started as a bucket-B item)
+  or verify manually this once.
+- **C — needs the release owner**: real provider credentials, real TTS podcast
+  generation, visual/UX judgment, and the final check of the pushed image.
+
+### 2. The image gate — test the artifact, not the repo
+
+A green suite on `main` is not a working image. Run:
 
 ```bash
-make docker-build-local   # 1. Build for the current platform, test locally
-make docker-push          # 2. Push version tags (does NOT update latest)
-                          # 3. Test the pushed version in staging/production
-make docker-push-latest   # 4. Promote: push version + update v1-latest
+make docker-build-local          # builds <version> + local tags
+make release-test TAG=<new> OLD_TAG=<previous>
 ```
 
+This runs two scenarios against real containers (`scripts/release-test/`):
+
+- **Fresh install**: empty DB → migrations on boot → in-image worker processes
+  a source → API/frontend/nginx-proxied checks.
+- **Upgrade**: boot the *published* previous image, seed data, swap to the new
+  image on the same volume → migrations apply, data survives.
+
+Caveat: `docker-build-local` tags with the current `pyproject.toml` version —
+`docker pull` the genuine previous tag before the upgrade test so you are not
+comparing the new build against itself.
+
+### 3. Fix loop with a re-test policy
+
+Findings become focused PRs through the normal review flow. After each merge:
+the cheap suite always re-runs; smoke/image gates re-run only if the fix
+touches what they cover; manual verification is not repeated unless the fix
+touches what was manually verified. Pre-existing bugs found along the way that
+are not release regressions become backlog issues instead of scope creep.
+
+## Cutting A Stable Release
+
+1. Confirm `main` is green and the confidence process above has run.
+2. Open the **cut PR**: bump `pyproject.toml`, date the `[Unreleased]` section
+   as `[<version>] - <date>`.
+3. After merge: `make tag`.
+4. Build and push version images **via CI** (it holds the registry
+   credentials): trigger the *Build and Release* workflow with
+   `push_latest=false`. Local `make docker-push` also works but requires
+   `docker login` on both registries.
+5. **Verify the pushed image** (bucket C, final gate): run it locally with
+   `make release-stack TAG=<version> [DUMP=<dev-data-dump>]` — a browsable,
+   isolated stack, optionally with a copy of real data — and walk the core
+   flows in the browser.
+6. Publish the GitHub release. A non-prerelease publication triggers the
+   workflow again and pushes the `v1-latest` tags automatically.
+7. Verify the `v1-latest` manifests on Docker Hub and GHCR (both arches, both
+   variants), and mark shipped issues with `released`.
+
+## Communication
+
+Release notes follow this structure (see v1.11.0 as the reference):
+
+1. One-line verdict + upgrade recommendation.
+2. Sections: Security, Features, Performance, Notable fixes.
+3. **Behavior changes for self-hosters** — anything that can require a config
+   tweak on upgrade gets an explicit callout.
+4. **Thanks** — credit every contributor by handle with what they shipped
+   (collect via `git log <last-tag>..<tag>` + `gh pr view` for handles), plus
+   the issue reporters collectively. Never skip this section.
+
+Announce on Discord after `v1-latest` is live.
+
+## Retro
+
+Close every release by asking: what should improve in this process? Apply the
+accepted improvements immediately — update this document, the scripts under
+`scripts/release-test/`, and the decision log while the context is fresh.
+
+## Docker Image Publishing (reference)
+
 | Command | What it does | Updates latest? |
 |---------|--------------|-----------------|
 | `make docker-build-local` | Build for current platform only (tags `<version>` + `local`) | No registry push |
-| `make docker-push` | Push version tags to registries | ❌ No |
-| `make docker-push-latest` | Push version + update `v1-latest` | ✅ Yes |
-| `make docker-release` | Full release (same as docker-push-latest) | ✅ Yes |
+| CI *Build and Release* (`push_latest=false`) | Push version tags via CI credentials | ❌ No |
+| GitHub release published (non-prerelease) | CI pushes version + `v1-latest` | ✅ Yes |
+| `make docker-push` / `docker-push-latest` | Local equivalents (need `docker login`) | ❌ / ✅ |
 | `make tag` | Create and push a git tag matching `pyproject.toml` | — |
 
-Publishing details:
-
 - **Platforms:** `linux/amd64`, `linux/arm64`
 - **Registries:** Docker Hub + GitHub Container Registry
 - **Image variants:** regular + single-container (`-single`)
 - **Version source:** `pyproject.toml`
 - Build issues: `docker builder prune`, then `make docker-buildx-reset`
 
-## Manual Verification
-
-Before publishing a stable release, manually verify the areas touched by the
-release. At minimum, cover:
-
-- installation or upgrade path changed by the release;
-- source ingestion and processing when content or worker behavior changed;
-- chat, search, notes, and notebooks when user workflows changed;
-- provider setup when model, credential, or API behavior changed;
-- Docker image startup when packaging, environment, or dependency changes
-  landed.
-
-Automated checks should catch regressions where possible, but the release owner
-chooses the manual matrix from the actual changes in the release.
+## Known Gotchas
+
+- **RC stack on non-default ports needs `API_URL`** or the browser talks to
+  `host:5055` — on a dev machine that is the development API (data crossover).
+  `rc-stack.sh` sets it; remember this for any custom setup.
+- **Containerized app + host services**: credentials pointing at local
+  services (Ollama, LM Studio) need `http://host.docker.internal:<port>`.
+- **SurrealDB import**: `OVERWRITE` goes after the type keyword
+  (`DEFINE FIELD OVERWRITE …`), and the exporter can leak a log line into the
+  dump — `rc-stack.sh` handles both.
+- **Multiple local SurrealDB instances**: check which one the dev `.env`
+  actually points at (`SURREAL_URL`) before exporting data.

CHANGELOG.md

@@ -5,6 +5,11 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Added
+- Release confidence process, documented and executable: `.github/RELEASE_PROCESS.md` now covers the risk-based test matrix (buckets A/B/C), the Docker image gate, the fix-loop re-test policy and the communication/credits/retro structure, backed by a new decision record (ADR-005) and versioned tooling under `scripts/release-test/` — `make release-test TAG= OLD_TAG=` runs fresh-install + upgrade scenarios against real images, and `make release-stack TAG= [DUMP=]` boots a browsable, isolated release-candidate stack (optionally with a copy of dev data) for manual verification
+
 ## [1.11.0] - 2026-07-11
 
 ### Added

Makefile

@@ -1,6 +1,7 @@
 .PHONY: run frontend check ruff database lint api start-all stop-all status clean-cache worker worker-start worker-stop worker-restart
 .PHONY: docker-buildx-prepare docker-buildx-clean docker-buildx-reset
 .PHONY: docker-push docker-push-latest docker-release docker-build-local tag export-docs
+.PHONY: release-test release-stack release-stack-down
 
 # Get version from pyproject.toml
 VERSION := $(shell grep -m1 version pyproject.toml | cut -d'"' -f2)
@@ -43,6 +44,25 @@ docker-buildx-clean:
 docker-buildx-reset: docker-buildx-clean docker-buildx-prepare
 	@echo "✅ Buildx reset complete!"
 
+# === Release Testing (see .github/RELEASE_PROCESS.md) ===
+
+# Automated image gate: fresh install + upgrade against real images.
+# Usage: make release-test TAG=1.12.0 OLD_TAG=1.11.0
+release-test:
+	@test -n "$(TAG)" || (echo "usage: make release-test TAG=<new> [OLD_TAG=<previous>]"; exit 1)
+	bash scripts/release-test/release-image-test.sh all \
+		"$(DOCKERHUB_IMAGE):$(TAG)" \
+		$(if $(OLD_TAG),"$(DOCKERHUB_IMAGE):$(OLD_TAG)")
+
+# Browsable RC stack for manual verification (optionally with a data dump).
+# Usage: make release-stack TAG=1.12.0 [DUMP=/tmp/dev-dump.surql]
+release-stack:
+	@test -n "$(TAG)" || (echo "usage: make release-stack TAG=<tag> [DUMP=<dump.surql>]"; exit 1)
+	bash scripts/release-test/rc-stack.sh up "$(TAG)" $(DUMP)
+
+release-stack-down:
+	bash scripts/release-test/rc-stack.sh down "$(or $(TAG),unused)"
+
 # === Docker Build Targets ===
 
 # Build production image for local platform only (no push)

docs/7-DEVELOPMENT/decisions/ADR-005-release-confidence-process.md

@@ -0,0 +1,69 @@
+# ADR-005: Releases pass a risk-based confidence process, gated on the real image
+
+- **Status**: Accepted
+- **Date**: 2026-07 (established during the v1.11.0 release)
+- **Related**: [RELEASE_PROCESS.md](../../../.github/RELEASE_PROCESS.md), `scripts/release-test/`, [ADR-004](ADR-004-background-workers.md)
+
+## Context
+
+Releases grew from a handful of fixes to 50+ commits spanning security
+hardening, features, migrations and dependency changes. Verification was
+ad-hoc: a green test suite on `main` plus whatever manual checks the release
+owner remembered. v1.11.0 proved the gap empirically — the unit suite was
+fully green while `sort_by=title` returned a 500 (a SEARCH-index interaction
+only a real SurrealDB exhibits) and clearing credential fields silently
+no-oped (two mirror-image bugs, frontend and API, that only an end-to-end
+path reveals). Neither class of bug is catchable by mocked tests, and neither
+was: both were found by the process this record establishes.
+
+## Decision
+
+**Every stable release passes a risk-based confidence process before cutting,
+and the final gate runs against the built Docker image — the artifact users
+receive — not the repository.**
+
+The process (mechanics in [RELEASE_PROCESS.md](../../../.github/RELEASE_PROCESS.md)):
+
+1. **Changelog audit first** — the release diff, fully represented in the
+   CHANGELOG, is the input for both the test plan and the communication.
+2. **Risk matrix over test list** — each change is classified by what it can
+   break and for whom, then assigned to a bucket: **A** (automated now),
+   **B** (automatable with investment — build the muscle when it compounds),
+   **C** (release-owner judgment: real credentials, real TTS, UX, the pushed
+   image). Security changes are probed for the inverse risk: does the
+   protection break legitimate use?
+3. **The image gate** — fresh-install and upgrade-with-data scenarios run
+   against real containers (`make release-test`), because packaging bugs
+   (supervisord flags, uv sync modes, migration ordering) never appear in the
+   suite.
+4. **Fix loop with a re-test policy** — findings become focused PRs; what
+   re-runs after each merge is defined up front. Pre-existing bugs that are
+   not release regressions become backlog issues, not scope creep.
+5. **Human gates stay human** — the pushed-image verification and the release
+   publication require the release owner explicitly; automation prepares,
+   people pull the trigger.
+6. **Retro closes the loop** — accepted improvements are applied to the
+   process docs and scripts in the same session.
+
+## Alternatives considered
+
+- **Keep ad-hoc verification** — free, but v1.11.0 showed it misses exactly
+  the bug classes that hurt users most (integration and packaging).
+- **Full CI-based E2E on every PR** — highest coverage, but a real
+  SurrealDB + worker + image build pipeline on every PR is slow and expensive;
+  the release boundary is where artifact-level confidence pays off.
+- **Community soak (RC tags)** — previously abandoned: slow feedback and low
+  participation; a deliberate confidence process front-loads what soaking
+  found late.
+
+## Consequences
+
+- Cutting a release costs hours, not minutes — deliberately: the cost scales
+  with release size, which is the point of the risk matrix.
+- Release muscle is versioned in-repo (`scripts/release-test/`, make targets)
+  and compounds: bucket-B investments from one release become bucket-A
+  automation for the next.
+- The upgrade scenario requires published previous images to remain available
+  on the registries.
+- The process assumes a release owner in the loop for buckets B/C — it is a
+  confidence process, not full automation.

scripts/release-test/docker-compose.release-test.yml

@@ -0,0 +1,43 @@
+services:
+  surrealdb:
+    image: surrealdb/surrealdb:v2
+    command: start --log info --user root --pass root rocksdb:/mydata/test.db
+    user: root
+    volumes:
+      - ${DATA_DIR:?set DATA_DIR}/surreal:/mydata
+
+  app:
+    image: ${APP_IMAGE:?set APP_IMAGE}
+    ports:
+      - "127.0.0.1:${API_PORT:?set API_PORT}:5055"
+      - "127.0.0.1:${FE_PORT:?set FE_PORT}:8502"
+    environment:
+      # Without an explicit API_URL the frontend's /config points the BROWSER
+      # at host:5055 — on a dev machine that is the development API, not this
+      # stack (silent data crossover between environments!)
+      - API_URL=${RC_API_URL:-}
+      - OPEN_NOTEBOOK_ENCRYPTION_KEY=${RC_ENCRYPTION_KEY:-release-test-key}
+      - SURREAL_URL=ws://surrealdb:8000/rpc
+      - SURREAL_USER=root
+      - SURREAL_PASSWORD=root
+      - SURREAL_NAMESPACE=${RC_SURREAL_NS:-open_notebook}
+      - SURREAL_DATABASE=${RC_SURREAL_DB:-open_notebook}
+    volumes:
+      - ${DATA_DIR}/notebook:/app/data
+    # Reach host services (e.g. Ollama on localhost:11434) via
+    # host.docker.internal on Linux too; Docker Desktop resolves it natively.
+    # Credentials pointing at local services must use
+    # http://host.docker.internal:<port> when the app runs in a container.
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    depends_on:
+      - surrealdb
+
+  proxy:
+    image: nginx:alpine
+    ports:
+      - "127.0.0.1:${PROXY_PORT:?set PROXY_PORT}:80"
+    volumes:
+      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
+    depends_on:
+      - app

scripts/release-test/nginx.conf

@@ -0,0 +1,18 @@
+# Reverse proxy in front of the frontend with default buffering ON — the SSE
+# streaming fix must work anyway because the API sends X-Accel-Buffering: no,
+# which nginx honors. Host is passed through verbatim to exercise the
+# frontend's Host validation with a real proxy in the path.
+server {
+    listen 80;
+    server_name _;
+
+    location / {
+        proxy_pass http://app:8502;
+        proxy_http_version 1.1;
+        proxy_set_header Host $http_host;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header Upgrade $http_upgrade;
+        proxy_set_header Connection "upgrade";
+        proxy_read_timeout 300s;
+    }
+}