Skip to content

Commit c42fa4c

Browse files
rschlaeflirschlaefli
committed
test(evaluation): add judge-based E3/E4/E7 eval dimensions and nightly CI
Extends the manage-assistant harness with grounding (E3, soft 0.90), proposal quality (E4, hard schema gate + soft judge 0.85), and degradation recovery (E7, hard no-fabrication gate + soft graceful 0.90) on the same live-turn mechanics, plus a guarded nightly workflow (schedule + workflow_dispatch; judge secrets absent -> clean skip, target unreachable -> hard failure via MANAGE_ASSISTANT_EVAL_REQUIRE_LIVE, zero recorded cases -> failure, never a green run that measured nothing). The printed OVERALL verdict is three-state: FAIL beats INCOMPLETE beats PASS, and PASS requires all eight dimensions measured. Readiness probes now refuse to run when base courses or the E6 payload elements are missing, so a half-seeded DB reads as an environment error instead of a model regression (see docs/solutions). Full-strength live run: E1 1.000 (12/12), E5 1.000 (24/24 trials), E6 1.000 (30/30 trials).
1 parent 5bace13 commit c42fa4c

36 files changed

Lines changed: 2450 additions & 43 deletions

‎.github/workflows/test-manage-assistant-eval-nightly.yml‎

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
name: Nightly manage-assistant eval (judge-based dimensions)
2+
3+
# Runs the manage-assistant eval harness (evaluation/manage-assistant, see
4+
# its README) against a live, already-running apps/chat deployment. Unlike
5+
# every other workflow in this directory, this one deliberately does NOT
6+
# spin up Postgres/mcp-lecturer/chat service containers from scratch --
7+
# those apps are only ever built into Docker images and deployed elsewhere
8+
# in this repo's existing CI (see v3_chat-*.yml, v3_mcp-lecturer-*.yml),
9+
# never booted standalone inside a GitHub-hosted runner. This workflow
10+
# targets whichever reachable deployment the MANAGE_ASSISTANT_EVAL_* secrets
11+
# point at (e.g. a staging environment) -- it never provisions or tears down
12+
# any environment itself.
13+
#
14+
# Nightly + workflow_dispatch only, deliberately NOT on push/pull_request:
15+
# this exercises the judge-based E3 (grounding)/E4 (proposal quality)/E7
16+
# (degradation recovery) dimensions added on top of the judge-free E1/E5/E6
17+
# harness, spends real judge-model and app-model inference budget every run,
18+
# and is gated on a live external deployment being reachable -- none of
19+
# which belong on the PR hot path.
20+
#
21+
# Guard (skip cleanly, never fail red and never pass by vacuity, when the
22+
# judge is not configured): a separate `check-judge-secret` job fails
23+
# nothing and runs no tests -- it only inspects whether
24+
# MANAGE_ASSISTANT_EVAL_JUDGE_MODEL/_API_KEY are configured as repository
25+
# secrets and writes a job-summary explanation either way. The `eval` job
26+
# below only runs when both are present (`if: needs.check-judge-secret
27+
# .outputs.configured == 'true'`), so an unconfigured judge shows as a
28+
# cleanly SKIPPED job, not a failure and not a silent no-op. All secret
29+
# values are referenced by name only (`secrets.<NAME>`); none is ever
30+
# inlined here.
31+
#
32+
# Note the deliberate asymmetry between the two ways this workflow can end
33+
# up not measuring anything:
34+
#
35+
# * judge not configured -> clean SKIP (the guard job above). This is
36+
# a known, declared state of the repo.
37+
# * judge configured but the
38+
# target is unreachable -> hard FAILURE, via
39+
# MANAGE_ASSISTANT_EVAL_REQUIRE_LIVE below.
40+
#
41+
# The harness's default local behavior is to skip cleanly (exit 0) when the
42+
# stack or DB is down, which is right for a developer but wrong here: it would
43+
# make a nightly that measured NOTHING report green, and a green safety gate
44+
# nobody can distinguish from "all dimensions passed" is worse than no gate.
45+
# `MANAGE_ASSISTANT_EVAL_REQUIRE_LIVE` inverts that default for this workflow
46+
# only (see tests/conftest.py).
47+
48+
on:
49+
schedule:
50+
- cron: '34 3 * * *'
51+
workflow_dispatch: {}
52+
53+
concurrency:
54+
group: ${{ github.workflow }}
55+
cancel-in-progress: true
56+
57+
jobs:
58+
check-judge-secret:
59+
runs-on: ubuntu-latest
60+
permissions: {}
61+
outputs:
62+
configured: ${{ steps.check.outputs.configured }}
63+
steps:
64+
- name: Check whether the judge credential is configured
65+
id: check
66+
env:
67+
JUDGE_MODEL: ${{ secrets.MANAGE_ASSISTANT_EVAL_JUDGE_MODEL }}
68+
JUDGE_API_KEY: ${{ secrets.MANAGE_ASSISTANT_EVAL_JUDGE_API_KEY }}
69+
run: |
70+
set -euo pipefail
71+
if [ -n "${JUDGE_MODEL}" ] && [ -n "${JUDGE_API_KEY}" ]; then
72+
echo "configured=true" >> "$GITHUB_OUTPUT"
73+
{
74+
echo "### Nightly manage-assistant eval: judge configured"
75+
echo ""
76+
echo "Both \`MANAGE_ASSISTANT_EVAL_JUDGE_MODEL\` and \`MANAGE_ASSISTANT_EVAL_JUDGE_API_KEY\` are set -- proceeding to the live eval job."
77+
} >> "$GITHUB_STEP_SUMMARY"
78+
else
79+
echo "configured=false" >> "$GITHUB_OUTPUT"
80+
{
81+
echo "### Nightly manage-assistant eval: SKIPPED"
82+
echo ""
83+
echo "The judge model used to score E3 (grounding), E4 (proposal quality), and E7"
84+
echo "(degradation recovery) is not configured, so the live eval job did not run --"
85+
echo "this is a clean skip, not a failure."
86+
echo ""
87+
echo "Missing repository secret(s):"
88+
if [ -z "${JUDGE_MODEL}" ]; then
89+
echo "- \`MANAGE_ASSISTANT_EVAL_JUDGE_MODEL\`"
90+
fi
91+
if [ -z "${JUDGE_API_KEY}" ]; then
92+
echo "- \`MANAGE_ASSISTANT_EVAL_JUDGE_API_KEY\`"
93+
fi
94+
echo ""
95+
echo "See evaluation/manage-assistant/README.md (\"Environment variables\") for the full list this harness reads, and src/manage_assistant_eval/judge.py for why these two specifically gate every judge-based check."
96+
} >> "$GITHUB_STEP_SUMMARY"
97+
fi
98+
99+
eval:
100+
needs: check-judge-secret
101+
if: needs.check-judge-secret.outputs.configured == 'true'
102+
runs-on: ubuntu-latest
103+
timeout-minutes: 30
104+
permissions:
105+
contents: read
106+
107+
steps:
108+
- name: Checkout repository
109+
uses: actions/checkout@v4
110+
111+
- name: Install uv
112+
uses: astral-sh/setup-uv@v7
113+
with:
114+
working-directory: evaluation/manage-assistant
115+
python-version: '3.12'
116+
enable-cache: true
117+
118+
- name: Install harness dependencies
119+
working-directory: evaluation/manage-assistant
120+
run: uv sync --locked
121+
122+
# Idempotent: creates the E6 indirect-injection collaborator fixtures
123+
# if they don't already exist on the target DB, no-ops otherwise (see
124+
# README "Seeding"). Safe to run every night.
125+
# PYTHONPATH=src is required, not decorative: pyproject.toml sets
126+
# `[tool.uv] package = false`, so the src/ layout is never installed into
127+
# the venv, and its `pythonpath = ["src"]` lives under
128+
# `[tool.pytest.ini_options]` -- which pytest honors and a bare
129+
# `python -m` does not. Without this the step dies on
130+
# `ModuleNotFoundError: No module named 'manage_assistant_eval'`.
131+
- name: Seed E6 indirect-injection fixtures
132+
working-directory: evaluation/manage-assistant
133+
run: PYTHONPATH=src uv run python -m manage_assistant_eval.seed --seed
134+
env:
135+
DATABASE_URL: ${{ secrets.MANAGE_ASSISTANT_EVAL_DATABASE_URL }}
136+
KLICKER_LECTURER_SUB: ${{ secrets.MANAGE_ASSISTANT_EVAL_LECTURER_SUB }}
137+
138+
# Runs the FULL suite (E1/E5/E6, judge-free, plus the new E3/E4/E7
139+
# judge-based dimensions) -- not just the new dimensions -- so a
140+
# nightly regression in the older judge-free checks is caught too.
141+
- name: Run manage-assistant eval suite
142+
working-directory: evaluation/manage-assistant
143+
run: uv run pytest -v
144+
env:
145+
# Turns "target unreachable" and "zero cases recorded" from a clean
146+
# skip into a failure, so this job can never go green without having
147+
# actually measured the dimensions. See this file's header.
148+
MANAGE_ASSISTANT_EVAL_REQUIRE_LIVE: '1'
149+
KLICKER_CHAT_BASE_URL: ${{ secrets.MANAGE_ASSISTANT_EVAL_CHAT_BASE_URL }}
150+
APP_SECRET: ${{ secrets.MANAGE_ASSISTANT_EVAL_APP_SECRET }}
151+
DATABASE_URL: ${{ secrets.MANAGE_ASSISTANT_EVAL_DATABASE_URL }}
152+
KLICKER_LECTURER_SUB: ${{ secrets.MANAGE_ASSISTANT_EVAL_LECTURER_SUB }}
153+
MANAGE_ASSISTANT_EVAL_JUDGE_MODEL: ${{ secrets.MANAGE_ASSISTANT_EVAL_JUDGE_MODEL }}
154+
MANAGE_ASSISTANT_EVAL_JUDGE_API_KEY: ${{ secrets.MANAGE_ASSISTANT_EVAL_JUDGE_API_KEY }}
155+
MANAGE_ASSISTANT_EVAL_JUDGE_API_BASE: ${{ secrets.MANAGE_ASSISTANT_EVAL_JUDGE_API_BASE }}
156+
157+
- name: Write job summary
158+
if: always()
159+
run: |
160+
{
161+
echo "### Nightly manage-assistant eval: ran"
162+
echo ""
163+
echo "Result: \`${{ job.status }}\`. Full per-dimension score summary (E1/E3/E4/E5/E6/E7) is in the \"Run manage-assistant eval suite\" step log above (\`ResultCollector.print_summary\`)."
164+
} >> "$GITHUB_STEP_SUMMARY"

‎docs/log.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# Log
22

3+
## 2026-07-27
4+
5+
- **Update**: [auth-model](./auth-model.md) records the Manage-assistant system prompt's new no-disclosure rule for the tool-output fence markers and sentinel, why it is not redundant with the fencing itself, and the before/after E6 measurement that motivated it.
6+
7+
- **New**: [solutions/best-practice/dev-seed-is-not-idempotent-reset-first](./solutions/best-practice/dev-seed-is-not-idempotent-reset-first.md) — `seed:raw` fails `P2002` on `Account` against an already-seeded DB _after_ its delete phase, leaving a half-seeded database; reset first, and seed harness-owned elements after the base seed, never before.
8+
39
## 2026-07-26
410

511
- **Update**: [testing](./testing.md) documents the lecturer MCP's `smoke:negative` authZ/negative-path script alongside the existing `smoke:local` happy path, and the new `test-mcp-lecturer` CI workflow (Postgres-only: unit tests, migrate + `seed:test`, boot the built server, run both smokes).

‎docs/solutions/best-practice/dev-seed-is-not-idempotent-reset-first.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
---
2+
module: prisma-data
3+
date: 2026-07-27
4+
problem_type: best_practice
5+
severity: medium
6+
tags:
7+
- dev-seed
8+
- prisma
9+
- devcontainer
10+
- idempotency
11+
- eval-harness
12+
---
13+
14+
# The Dev Seed Is Not Idempotent — Reset Before Reseeding
15+
16+
## Context
17+
18+
`pnpm --filter @klicker-uzh/prisma-data run seed:raw` (`seed:test` +
19+
`seed:assessment-course`) is not safe to re-run on a database that already has
20+
base data. Against an already-seeded DB it fails partway with
21+
`P2002 UniqueConstraintViolation` on `modelName: 'Account'`
22+
(`packages/prisma-data/src/data/seedTEST.ts`) — but only _after_ its delete
23+
phase has already run.
24+
25+
The failure is therefore worse than a no-op: each attempt wipes existing rows
26+
and then dies before recreating the later ones, leaving a **half-seeded**
27+
database (elements present, zero courses). Retrying reproduces it exactly, so
28+
the state looks stable rather than broken, and `Element` rows deleted by the
29+
wipe phase do not come back.
30+
31+
This is easy to misread as an application bug. It surfaced as seven confident
32+
failures in the manage-assistant eval harness
33+
(`evaluation/manage-assistant`) that read exactly like model regressions —
34+
including four hard-gate prompt-injection "failures" — when the real cause was
35+
that the seed had left the DB with no courses and no injection-payload
36+
elements.
37+
38+
## Guidance
39+
40+
Do not re-run the seed on top of existing data. Reset first, exactly as
41+
`.devcontainer/post-create.sh` does:
42+
43+
```bash
44+
pnpm --filter @klicker-uzh/prisma exec prisma migrate reset --skip-seed --force \
45+
&& pnpm --filter @klicker-uzh/prisma exec prisma db push
46+
pnpm --filter @klicker-uzh/prisma-data run seed:raw
47+
```
48+
49+
Use `seed:raw`, not `seed`, inside the devcontainer: `seed` goes through
50+
`util/_run_with_infisical.sh`, which needs `jq` and real secrets that the
51+
self-contained devcontainer deliberately does not have.
52+
53+
Two consequences worth remembering:
54+
55+
- **The base seed deletes `Element` rows.** Anything that seeds its own
56+
elements (the eval harness's E6 injection payloads, ad-hoc fixtures) must be
57+
seeded _after_ the base seed, never before, or it is silently destroyed while
58+
its owning `User` row survives — which makes readiness probes that only check
59+
for the user report a ready environment.
60+
- **Fixture-dependent test suites should assert their fixtures exist**, not
61+
assume them. A suite that blames the model (or the code) for a missing
62+
fixture is worse than one that refuses to run. `evaluation/manage-assistant`
63+
now checks both base courses and its own payload elements before running, for
64+
exactly this reason.

0 commit comments

Comments
 (0)