Skip to content

Commit 1c5f1c6

Browse files
authored
chore(release): automate container probes + document post-tag re-cut (#1198)
Retro improvements from the v1.14.0 release: - release-image-test.sh gains a `probe` scenario (part of `all`) that runs the two container-level checks done by hand during v1.14.0: OPEN_NOTEBOOK_WORKER_MAX_TASKS reaching the in-image worker (#1141's supervisord `sh -c` expansion), and the worker surviving startup with HTTP_PROXY set while the user's NO_PROXY value is preserved (#1160, internal SurrealDB websocket not tunneled). Self-contained standalone containers with their own cleanup, so a probe failure can't leak into the fresh/upgrade scenarios. - RELEASE_PROCESS.md documents two gotchas the release hit: never leave the version bump uncommitted on a branch (it leaked into an unrelated fix PR), and a post-tag fix requires a full re-cut (move tag + rebuild images), not just a tag nudge, or publication promotes the un-fixed artifact to v1-latest. Verified: `release-image-test.sh probe lfnovo/open_notebook:1.14.0` passes 4/4.
1 parent 30c7e2a commit 1c5f1c6

3 files changed

Lines changed: 103 additions & 2 deletions

File tree

.github/RELEASE_PROCESS.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,20 @@ accepted improvements immediately — update this document, the scripts under
148148

149149
## Known Gotchas
150150

151+
- **Never leave the version bump uncommitted on a branch.** Editing
152+
`pyproject.toml` / `CHANGELOG.md` for the cut and then switching branches
153+
carries those unstaged changes along, and the next `git add -A` sweeps the
154+
bump into an unrelated fix PR — the version silently ships inside a `fix(...)`
155+
commit. The cut is always the **last** step, on its own branch, committed
156+
immediately; if you must build an image early (which needs the bumped
157+
version), do it on a throwaway branch and `git stash`/discard before moving
158+
on (v1.14.0 lesson).
159+
- **A fix that lands after the cut requires a full re-cut, not a tag nudge.**
160+
If the release was tagged but not yet published (no GitHub release, no
161+
`v1-latest`) and a blocker is found in bucket C, the tag must move to the new
162+
commit AND the version images must be rebuilt — a stale tag or stale registry
163+
image will otherwise be what publication promotes to `v1-latest`. The exact
164+
sequence is in `runbook.md` → "Re-cut after a post-tag fix" (v1.14.0 lesson).
151165
- **RC stack on non-default ports needs `API_URL`** or the browser talks to
152166
`host:5055` — on a dev machine that is the development API (data crossover).
153167
`rc-stack.sh` sets it; remember this for any custom setup.

CHANGELOG.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,11 @@ All notable changes to this project will be documented in this file.
55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

8+
## [Unreleased]
9+
10+
### Changed
11+
- Release image gate gained a `probe` scenario (`make release-test` runs it as part of `all`): container-level checks that a Python test suite can't cover because they depend on the shipped image's process supervision — `OPEN_NOTEBOOK_WORKER_MAX_TASKS` reaching the in-image worker (the supervisord `sh -c` expansion), and the worker surviving startup with `HTTP_PROXY` set while a user's `NO_PROXY` value is preserved (the internal SurrealDB websocket not being tunneled). Both were manual probes during the v1.14.0 release; they now run automatically. Release-process docs gained the post-tag re-cut sequence and a note on never leaving the version bump uncommitted (v1.14.0 retro)
12+
813
## [1.14.0] - 2026-07-20
914

1015
### Added

scripts/release-test/release-image-test.sh

Lines changed: 84 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,16 @@
11
#!/bin/bash
22
# Release image gate: fresh-install and upgrade tests against built Docker images.
33
#
4-
# Usage: release-image-test.sh <fresh|upgrade|all> <new-image> [old-image]
4+
# Usage: release-image-test.sh <fresh|upgrade|probe|all> <new-image> [old-image]
55
# e.g. release-image-test.sh all lfnovo/open_notebook:1.12.0 lfnovo/open_notebook:1.11.0
66
#
7+
# Scenarios:
8+
# fresh - empty DB -> migrations on boot -> worker processes a source
9+
# upgrade - boot old image, seed, swap to new image on the same volume
10+
# probe - container-level checks for image-specific behavior (worker
11+
# concurrency env expansion, HTTP proxy / no_proxy handling)
12+
# all - all of the above
13+
#
714
# IMPORTANT: `make docker-build-local` tags the build with the CURRENT pyproject
815
# version. If that version matches a published release, `docker pull` the
916
# genuine old tag first or the upgrade test will compare the new build against
@@ -140,10 +147,85 @@ print('yes' if any(n.get('name')=='release-probe' for n in nbs) else 'no')" 2>/d
140147
echo
141148
}
142149

150+
# Container-level probes for release changes that a repo test suite cannot cover
151+
# because they depend on the shipped image's process supervision and env
152+
# handling (supervisord, entrypoint), not on Python. Each is self-contained
153+
# (standalone `docker run`, own cleanup) so a failure here can never leak state
154+
# into the fresh/upgrade scenarios.
155+
probe_test() {
156+
echo "═══ CONTAINER PROBES — $NEW_IMAGE"
157+
158+
# #1141: OPEN_NOTEBOOK_WORKER_MAX_TASKS must reach the worker. supervisord's
159+
# command= does not run through a shell, so the value is only honored if the
160+
# command is wrapped in `sh -c`. Boot with the var set and a dead DB (the
161+
# worker logs its configured concurrency before it ever connects), then read
162+
# the value back from its startup log.
163+
local CID
164+
CID=$(docker run -d --rm \
165+
-e OPEN_NOTEBOOK_WORKER_MAX_TASKS=2 \
166+
-e SURREAL_URL=ws://127.0.0.1:9/rpc \
167+
-e OPEN_NOTEBOOK_ENCRYPTION_KEY=probe \
168+
"$NEW_IMAGE" 2>/dev/null)
169+
if [ -z "$CID" ]; then
170+
bad "worker-concurrency probe: container did not start"
171+
else
172+
local CONC=""
173+
for i in $(seq 1 20); do
174+
CONC=$(docker logs "$CID" 2>&1 | grep -oiE "up to [0-9]+ concurrent tasks" | grep -oE "[0-9]+" | head -1)
175+
[ -n "$CONC" ] && break
176+
sleep 3
177+
done
178+
check "OPEN_NOTEBOOK_WORKER_MAX_TASKS honored by the in-image worker" "2" "$CONC"
179+
docker rm -f "$CID" >/dev/null 2>&1
180+
fi
181+
182+
# #1160: with HTTP_PROXY set, the internal SurrealDB websocket must NOT be
183+
# tunneled through the proxy (websockets 15 auto-detects proxy env even for
184+
# ws://). The app injects the internal hosts into NO_PROXY at startup. Boot
185+
# with a dead proxy set plus a user NO_PROXY value, against a real DB, and
186+
# assert the worker reaches RUNNING and the user's NO_PROXY value survives.
187+
local NET=onrelprobe-net
188+
docker network create "$NET" >/dev/null 2>&1
189+
docker run -d --rm --network "$NET" --name onrelprobe-surreal \
190+
surrealdb/surrealdb:v2 start --user root --pass root --bind 0.0.0.0:8000 memory >/dev/null 2>&1
191+
sleep 6
192+
local APP
193+
APP=$(docker run -d --rm --network "$NET" --name onrelprobe-app \
194+
-e SURREAL_URL=ws://onrelprobe-surreal:8000/rpc -e SURREAL_USER=root -e SURREAL_PASS=root \
195+
-e SURREAL_NAMESPACE=open_notebook -e SURREAL_DATABASE=probe \
196+
-e OPEN_NOTEBOOK_ENCRYPTION_KEY=probe \
197+
-e HTTP_PROXY=http://127.0.0.1:9 -e HTTPS_PROXY=http://127.0.0.1:9 \
198+
-e NO_PROXY=my-corp.example \
199+
"$NEW_IMAGE" 2>/dev/null)
200+
if [ -z "$APP" ]; then
201+
bad "no_proxy probe: container did not start"
202+
else
203+
local RUNNING="" MERGED=""
204+
for i in $(seq 1 20); do
205+
RUNNING=$(docker logs "$APP" 2>&1 | grep -c "worker entered RUNNING state")
206+
[ "$RUNNING" != "0" ] && break
207+
sleep 3
208+
done
209+
if [ "$RUNNING" != "0" ]; then ok "worker starts with HTTP_PROXY set (internal ws not tunneled)"; else
210+
bad "worker did not reach RUNNING with HTTP_PROXY set"; docker logs "$APP" 2>&1 | tail -8 | sed 's/^/ /'
211+
fi
212+
local REJECT
213+
REJECT=$(docker logs "$APP" 2>&1 | grep -ciE "403|proxy.*reject")
214+
check "no proxy-rejection (403) in worker/API startup" "0" "$REJECT"
215+
MERGED=$(docker exec "$APP" sh -c 'python -c "import os;print(\"my-corp.example\" in (os.environ.get(\"NO_PROXY\") or \"\"))"' 2>/dev/null)
216+
check "user NO_PROXY value preserved (not clobbered)" "True" "$MERGED"
217+
docker rm -f "$APP" >/dev/null 2>&1
218+
fi
219+
docker rm -f onrelprobe-surreal >/dev/null 2>&1
220+
docker network rm "$NET" >/dev/null 2>&1
221+
echo
222+
}
223+
143224
case "${1:-all}" in
144225
fresh) fresh_test ;;
145226
upgrade) upgrade_test ;;
146-
all) fresh_test; upgrade_test ;;
227+
probe) probe_test ;;
228+
all) fresh_test; upgrade_test; probe_test ;;
147229
esac
148230

149231
echo "═══ RESULT: $PASS passed, $FAIL failed"

0 commit comments

Comments
 (0)