fix(skills): correct nightly data gathering and lane scoring - #240
Draft
LukeAVanDrie wants to merge 1 commit into
Draft
fix(skills): correct nightly data gathering and lane scoring#240LukeAVanDrie wants to merge 1 commit into
LukeAVanDrie wants to merge 1 commit into
Conversation
LukeAVanDrie
force-pushed
the
nightly-gather-fixes
branch
4 times, most recently
from
August 14, 2026 18:09
a29d5e4 to
666432f
Compare
Author
|
Here is an example report that this skill generates now. Triggered via Nightly Status — 2026-08-1426 of 44 scheduled runs failed. Seven distinct causes. Two are new as of the night of 2026-08-13 and account for 12 lanes — both trace to image-pin changes merged that day. No coverage gaps. 1. Bad release image pins from #2260 — 5 lanes, new 2026-08-13
The sidecar is an init container (
Will not clear on a rerun. Wide EP GKE was already chronic for another reason; this is now its top blocker. 2. vLLM v0.26.0 modelserver crash-loops — 7 lanes, new 2026-08-13Same commit bumped Optimized Baseline (Intel XPU) · Optimized Baseline (CKS GPU) · Precise Prefix Cache Routing (CKS GPU) · Precise Prefix Cache Routing (Intel XPU) · Predicted Latency Routing (CKS GPU) · Tiered Prefix Cache CPU Offloading (Intel XPU) · Wide EP LWS (Intel XPU) Five of the seven were green on 2026-08-12. Optimized Baseline (CKS GPU) has been red since 08-05 for a different reason; last night's failure is this one. The harness collects events and 3. GKE TPU — DWS requests never fulfilled (4 lanes, chronic)Optimized Baseline (17, last green 07-28) · PD Disaggregation (16, 07-29) · Tiered Prefix Cache (5, 08-09) · Precise Prefix Cache (2 runs, never green — new lane). TPU capacity, not the guides. The stale namespace shows teardown is also not cleaning up. 4. Benchmark harness never completes (3 lanes, last green 2026-08-05)
5. GKE GPU cluster out of capacity (2 lanes)
6. CKS RDMA capacity (1 lane)
7. Four lanes broken since they were added
These need their authors, not tracking issues. Coverage GapsNone. Five lanes show
Highest-leverage fix: repin the two tags in |
The nightly skills are executable documentation: a session reads them and
runs their bash and jq against the GitHub Actions API to triage failures and
produce the weekly report. Both sampled a fraction of the fleet, scored the
lanes they did see against wrong denominators, and asked for report fields no
phase computed.
Gathering
gh run list --limit N truncates. llm-d/llm-d produced 915 scheduled runs
over 2026-08-06..12, so Step 1's --limit 50 covered about nine hours and
Phase 0's --limit 500 under five days. Raising it does not help: --limit and
the REST runs endpoint both stop at 1000 results and exit 0. A 7-day
server-side query returned 1000 of 1020, dropping 20 runs across 12 nightly
lanes. Both skills now page the REST endpoint one day per query and check
each day against the server's total_count.
Neither skill filtered on event, so PR and dispatch runs diluted pass rates.
The [Nn]ightly|[Ee]2[Ee] name regex also swept in 252 weekly runs of the
consolidate-status-* workflows, which are named "Past Status - ... E2E
(...)". Lane selection is now on .path with an explicit exclude list, and
event=schedule is applied server-side.
A lane that stops running is invisible to a check that starts from a list of
runs: it reads as not failing while testing nothing. A new phase anti-joins
the workflow registry against the runs, classifies each zero-run lane
REMOVED, STALE, NEVER RAN, DISABLED, TOO NEW or LOOKUP FAILED, and proposes
rename successors by token overlap.
Scoring
Phase 1 crashed with "Cannot index object with null" whenever a nightly was
in flight, which is most nights, because {...}[null] raises before // "?"
can apply. In-progress runs now render as ?.
Pass rate alone cannot separate a lane that broke last night from one that
passed early in the week and has been dead since; both read 3/7. Lanes now
classify on rate and trailing failure streak together. NO VERDICT fired only
when every run was cancelled, so one stray failure among seven cancellations
scored CHRONIC 0/1. It is now a majority test, exposed as a `red` field that
attribution, clustering and lifetime lookup all select on; each previously
selected on the streak alone and pulled in NO VERDICT lanes carrying a stale
failure reported as current. Lifetime streaks counted cancellations as
failures, turning a lane with 53 cancellations and 2 failures into a
55-night streak reported as the fleet's oldest breakage. They count decisive
runs only.
Week-over-week compared a 7-day window against an 8-day one, so a lane's
denominator changed between halves of the comparison. Both windows are now
$DAYS long and disjoint, and the block asserts it. An all-green week
produced two jq errors: `add` over an empty array is null.
Routing
Skill discovery scans .claude/skills/*/SKILL.md one level deep. All seven
sub-skills sit at .claude/skills/nightly/<name>/SKILL.md, a level below
that, so none was registered and the `name: nightly:<x>` in their
frontmatter declared an identity the loader never reads. A session reaching
for one got "no such skill" and improvised the phase from scratch. All 13
references are now file paths to Read.
Step 4 classified straight off the failed step name, but Standup and Run
each name a whole phase of the reusable workflow. On 2026-08-12 they covered
a pod-readiness timeout, DWS provisioning starvation, a malformed kustomize
path and a CrashLoopBackOff between them. Step 4b pulls each failing step's
real output first and groups lanes on error text.
Error text
gh run view --log renders every step as the literal string UNKNOWN STEP, and
the run-level log zip holds per-job files only. errtext.sh windows the job
log on the failing step's timestamps and cuts on the ##[group] boundaries,
taking 1700 lines to under ten. The download needs --allow-escape-sequences,
without which gh exits 1 on a body carrying ANSI and prints a header and
nothing else, indistinguishable from a step that produced no output.
Standup emits its own ::group:: around the env dump, so the cut lands inside
it and returns a content-free body on 9 of 34 runs. errtext.sh falls back to
errsig.sh, which greps the whole job log for error signatures, excluding
four post-failure debug patterns that appear in 29 of 41 failing job logs
and crowd the real cause out of the tail. All nine bodies then name a cause.
Report content
Computed the figures the template left as placeholders: fleet decisive-run
count, ex-chronic pass rate, never-passed count, mean failure minutes per
lane, under-merge detection in clustering, and lifetime streaks carrying a
first-run column so a lane merged yesterday does not qualify as
never-passed on a single run. Failure categories come from the step-ID
prefixes in reusable-ci-nightly-benchmark.yaml, replacing a five-category
table no phase produced. Categorizing by step name reported phantom
accelerator contention: "Request TPUs (DWS)" is infra_request_tpus and dies
in about a second on stale state.
Shared scripts
The shell and jq live in .claude/skills/nightly/scripts/ as 15 files with a
README index, invoked as sh $NS/<name>.sh. Both documents need most of the
same code, and duplicating it as heredocs let the copies drift: the TOO NEW
guard reached one document and not the other, so the weekly report counted
every lane added yesterday as a coverage gap. Scripts that fan out feed
xargs one NUL-delimited record and split the fields with cut, because xargs
splits on any whitespace and a lane display name shifts every field of that
row and of the rows batched after it.
Reference tables
llm-d-workload-variant-autoscaler has no scheduled E2E of its own; those
lanes live in llm-d/llm-d as nightly-e2e-workload-autoscaling-*.
llm-d-infra's own three nightlies were unmonitored. The
reusable-*-helmfile.yaml workflows named in the table do not exist, and the
same stale names appeared in .claude/CLAUDE.md, which loads into every
session in this repo. reusable-ci-nightly-benchmark.yaml is not
workflow_call-only: it carries `schedule: - cron: '0 0 * * *'` and runs in
llm-d-infra in its own right. The schedule matrix is derived from the crons.
The session shell is zsh, where "/$cand[.]y" parses as an array subscript and
raises "bad floating point constant", and `set -- $pair` does not word-split.
Both are named at the top of the router.
Every bash and jq block in both files ran in order against llm-d/llm-d and
llm-d/llm-d-infra, including the empty, all-green, in-flight, failed-download
and month-boundary paths.
Signed-off-by: Luke Van Drie <lukevandrie@google.com>
LukeAVanDrie
force-pushed
the
nightly-gather-fixes
branch
from
August 14, 2026 21:19
666432f to
5076fd1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Corrects the bash and jq in
.claude/skills/nightly/.nightly/SKILL.mdtriages a failing nightly run andnightly/report/SKILL.mdproduces the weekly fleet report. An agent runs their code blocks verbatim against the GitHub Actions API, so a wrong query returns a wrong report without raising an error. Only.claude/changes; CI behavior is unchanged.Sub-skill reachability. Skill discovery scans
.claude/skills/*/SKILL.mdone level deep, so the seven sub-skills at.claude/skills/nightly/<name>/SKILL.mdwere never registered andSkill(nightly:<x>)failed. All 13 references in the router and in.claude/CLAUDE.mdbecome file paths to Read.Data gathering.
gh run list --limit Nsampled the fleet.llm-d/llm-dserved 915 scheduled runs over 2026-08-06..12, so--limit 500covered under five days, and both--limitand the REST endpoint stop at 1000 results while exiting 0. Queries page the REST endpoint with server-sideeventandcreatedfilters, one day per query, and check each day's returned count against the server'stotal_count. Lane selection moves to.pathwith an exclude list, which keeps theconsolidate-status-*workflows out of the fleet count. The daily triage window is a rolling 24-hour timestamp; a bare date spanned two nights and doubled every lane.Coverage gaps. A new phase anti-joins the workflow registry against the runs, so a lane that stopped running no longer reads as healthy. Two guards keep the result honest. Lanes registered within the last two days are marked TOO NEW, since they have not reached a first cron. A token-overlap matcher proposes rename successors, because the
-acc-migration left no shared prefixes to match on. Candidates are filtered by the same exclude list as lane selection; unfiltered,consolidate-status-<lane>carries the whole lane name plus two tokens and wins the overlap on every row. The matcher emits its top two candidates, because token overlap ties on platform synonyms andibm-is the OpenShift-GPU prefix.Triage on error text.
StandupandRuneach cover several unrelated causes, so classifying on the step name grouped failures wrongly. Step 4b pulls the failing step's output:errtext.shwindows the job log on the step's timestamps, then falls back toerrsig.sh, which greps the whole log through a noise filter, when the window names no cause. Step 4c adds a trailing failure streak and last-green date per lane. Past roughly 15 red lanes, an Explore subagent summarizes the collected error text so it stays out of the conversation context.Lane scoring. Lanes classify on pass rate and trailing streak together. NO VERDICT is a majority-of-cancellations test, and those lanes stay out of attribution and clustering. Lifetime streaks count decisive runs only, and report
NEVER PASSED,NO DECISIVE RUNorLOOKUP FAILEDrather than dropping a lane out of the table.Report content. Figures the template left as placeholders are computed: fleet summary, lifetime table, failure categories derived from the step-ID prefixes in
reusable-ci-nightly-benchmark.yaml, per-lane failure cost, and under-merge detection in clustering.Shared scripts. The shell and jq live in
.claude/skills/nightly/scripts/as 15 files with aREADME.mdindex, invoked assh $NS/<name>.sh. Both documents need most of the same code, and duplicating it as heredocs let the copies drift: the TOO NEW guard reached one document and not the other, so the weekly report counted every lane added yesterday as a coverage gap. Scripts that fan out feedxargsone NUL-delimited record and split the fields withcut, becausexargssplits on any whitespace and a lane display name likeNightly - Optimized Baseline E2E (GKE GPU)shifts every field of that row and of the rows batched after it.Reference tables.
llm-d-workload-variant-autoscalernightlies run inllm-d/llm-d.llm-d-infra's own three nightlies were unmonitored. Thereusable-*-helmfile.yamlentries in the router and in.claude/CLAUDE.mdnamed workflows that do not exist. The schedule matrix is derived from the crons instead of hand-maintained.Why is this change needed?
Each defect fails silently. Truncation drops the oldest runs and exits 0; a 7-day query returned 1000 of 1020 runs. A removed lane contributes no runs and reads as healthy. An unregistered sub-skill raises no error, so sessions improvised the missing phase.
How was this tested?
Every bash and jq block in both files ran in order against
llm-d/llm-dandllm-d/llm-d-infrawith exit 0: 278 scheduled runs over 7/7 days forllm-d, 42 lanes classified, 5 REMOVED gaps each proposing a running-acc-successor, 24 failures attributed to their steps, 27 lifetime rows with no stderr.Edge cases exercised: the truncation guard on a >1000-run range, which exits 1 with the split-by-hour message;
errtext.shon failed runs and on a run with no failing job; a workflow basename the API 404s on; a lane whose every run was cancelled; the Phase 6 window check for equal length and no overlap; an all-green input; an in-flight run; a failed log download; a month-boundary window.The failure-category taxonomy was cross-checked against the badge commit messages on
gh-pages.Checklist
git commit -s) per DCOmake test) — n/a, no Makefile in this repomake lint) — n/a, no linter covers.claude/