Harden traceability + self-learning tail: deterministic graph assembly, safe SDTM staging, gated skill-PR

[Griphu]

Summary

Hardens the tail of the USDM→TFL pipeline by moving correctness-critical, deterministic work out of LLM agent steps and into pinned, tested Python scripts, gating the self-learning loop so a clean run doesn't pay for it, and making the skill-lesson PR non-fatal. Net: the pipeline grows from 13 to 15 steps, but the two new steps are cheap deterministic scripts that remove non-determinism and cost from the parts of the flow where an LLM added risk, not value.

All changes came out of a critical review of the workflow against the Mediforce workflow-authoring golden rules (executor-choice / "is that really an agent step?"). Each is independently motivated below.

Why these changes

F1b — Split the traceability graph assembly out of the agent (correctness)

build-traceability was a single ~25-min agent that both assembled the objective→endpoint→SDTM→ADaM→TLF graph (a pure join over five JSON artifacts + coverage math + the issues feed) and rendered the interactive HTML explorer. A traceability graph assembled by an LLM can silently drop an edge or miscount endpoint coverage — which defeats the entire purpose of the artifact ("prove with the numbers").

• New assemble-trace-graph script step (container/build_trace_graph.py) does the join, two-way coverage, absent-domain detection, status tallies, and issues feed deterministically in code, emitting trace_graph.json + manifest.json per the traceability-builder graph-data-schema.md.
• build-traceability is now render-only — it reads the authoritative trace_graph.json and produces the HTML without recomputing anything. The traceability-builder skill gained an explicit render vs assemble mode note.
• The rich interactive HTML (browser-untestable here) stays an agent job; the numbers no longer do.

F2 — De-risk SDTM staging in plan-tlfs (reliability)

plan-tlfs mixed a deterministic file operation — identify the USDM among the uploads and copy up to 200 SDTM files from /data/ into /workspace/sdtm/ — into the same LLM step that plans TLFs. An agent hand-copying files can mis-identify the USDM or silently drop datasets.

• New baked container/stage_inputs.py content-detects the USDM (by usdmVersion / study.versions, so a Dataset-JSON SDTM file is never mistaken for it), writes it to /workspace/usdm.json, persists every other upload into /workspace/sdtm/, and fails fast if no USDM or no SDTM is present.
• plan-tlfs now runs it as a mandatory STEP 0 (python3 /app/container/stage_inputs.py) before planning; the tlf-planner skill documents the same. The LLM triggers one reliable tool instead of hand-copying files.
• Note: this stays inside the plan-tlfs agent container on purpose — uploaded files are mounted at /data only on the step that directly follows the upload, and that mount is agent-only.

F3 — Make open-skill-pr fail-soft (robustness)

open-skill-pr is a script step, and continueOnError is honoured only on action steps — so a GitHub hiccup or a token missing pull-requests:write would fail the entire clinical pipeline at its very last step, after all TLFs were generated and human-approved, over a non-critical skill-improvement side effect.

• open_skill_pr.py's fail() now records {prCreated:false, reason} and exits 0 instead of 1, so the run advances to done. The reason is still surfaced in result.json + stderr for the step log.

F7 — Gate the self-learning tail (efficiency)

On a clean first-pass run (no revisions), the ~15-min propose-skill-update agent and open-skill-pr still ran, just to produce a no-op.

• New check-feedback script step (container/check_feedback.py) reads /workspace/review_feedback.jsonl and emits {hadRevisions}. A conditional branch (output.hadRevisions == true/false, the proven validation-gate pattern) routes into propose-skill-update → open-skill-pr only when a review actually requested changes; otherwise the run goes straight to done.

F8 — README accuracy

The README claimed fixtures/ was baked into the image, but the Dockerfile only COPY container/. Fixed the overclaim, documented all four baked container/ scripts, and rewrote the pipeline table (now 15 steps) + the traceability/self-learning descriptions.

Drive-by: fixed pre-existing broken tests

tests/test_open_skill_pr.py referenced LESSONS_FILE and a 2-arg append_lesson that no longer existed after an earlier multi-skill refactor (2 of 3 tests were failing on main). Updated to the current lessons_file_for(skill) / 3-arg API and added a fail-soft assertion for F3.

Routing changes (src/cdisc-case-3.wd.json)

… → review-tlfs --approve--> assemble-trace-graph (NEW, script)
                              → build-traceability (now render-only)
                              → check-feedback (NEW, script)
                                 ├─ hadRevisions=true  → propose-skill-update → open-skill-pr → done
                                 └─ hadRevisions=false → done

Validation

• Schema: mediforce workflow register --dry-run → OK (15 steps, 18 transitions).
• Build context (Phase 3a): dockerfile: Dockerfile resolves at repo root; COPY container/ present; every step command path lands at /app/container/.
• Syntax (Phase 3b): all four scripts py_compile clean.
• Behavior (Phase 3c): 12/12 python tests/run_tests.py — new tests for build_trace_graph.py (join/coverage/absent-domain/issues/embedded artifacts), stage_inputs.py (USDM detection incl. Dataset-JSON, fail-fast paths), check_feedback.py (hadRevisions gate), plus the repaired open_skill_pr.py tests. Fixtures persisted under tests/fixtures/.

Not runtime-verified here (need an actual run): the Docker image build itself, the render-only agent's HTML output, and end-to-end execution on the platform.

Pinning

Build-context commit fields (11 in the .wd.json + externalSkillsRepo.commit) are pinned to 4163e27, which contains the new container/ scripts; HEAD (2aef716) carries the filled .wd.json. 4163e27 is reachable from HEAD, so the image builds from a commit that has the scripts while the definition registers from the working tree — the standard decoupled two-SHA shape.

🤖 Generated with Claude Code