fix(link-check): count broken links from the errors section only by JOhnsonKC201 · Pull Request #2080 · harvard-edge/cs249r_book

JOhnsonKC201 · 2026-08-17T23:58:45Z

Summary

The nightly link-rot summary counts broken links by scanning the whole lychee report for lines shaped * [<status>] <url>. That matches the wrong set in both directions, so the counts in #1810 are wrong on five of the nine sites. This scopes the scan to the report's "Errors per input" section and accepts non-numeric statuses.

Area

Book (textbook content, figures, exercises)
TinyTorch (modules, tests, milestones)
StaffML (interview questions, challenges)
Kits (hardware labs)
Infrastructure (CI/CD, scripts, config)

Changes

Measured against last night's sweep, run 31995703306:

Site	lychee errors + timeouts	tracker reported	after
Book	2	`?`	2
Labs	0	0	0
Kits	0	1	0
MLSys·im	0	`?`	0
Slides	2	`?`	2
Instructors	35	36	35
TinyTorch	1	27	1
Unified Site	1	3	1
StaffML	0	0	0

Two separate causes, same line of code.

Over-counting. The report also carries a ## Redirects per input section whose entries have the identical * [<status>] <url> shape, so every followed redirect was counted as broken. TinyTorch is the clearest case: lychee found 1 error, the tracker said 27, and 26 of those were redirect entries like

* [200] <https://www.nature.com/articles/323533a0> | Redirect: Followed 3 redirects resolving to the final status of: OK.

Four of them were [403], which is a status this workflow already passes --accept 200,403 for. Links lychee was explicitly told to accept were still being reported broken.

Under-counting. The status is not always numeric. A request that never produced an HTTP response is reported as [ERROR] or [TIMEOUT], covering TLS failures, DNS failures, timeouts and missing local files. [[:digit:]]+ cannot match those, so they were dropped from both the count and the triage sample. When every failure on a site was that kind, the count parsed to 0 and the "broken but unknown" guard rewrote it to ?. That is why Book, MLSys·im and Slides show ? in #1810 with no URL list, and it means the two failures that are hardest to diagnose from a URL alone were the ones being hidden:

* [ERROR]   <https://web.eng.fiu.edu/.../563900a043.pdf> | SSL certificate not trusted
* [ERROR]   <file:///.../slides/tinyml/chapter-3-applications-of-tinyml> | Cannot find file
* [TIMEOUT] <https://larissasuzuki.com/> | Timeout

The fix replaces the two whole-file greps with one awk pass scoped to the errors section, matching any bracketed status. It also strips lychee's markdown autolink delimiters, so the tracker lists https://example.com rather than <https://example.com> as it does today.

The step's contract is unchanged: same three outputs, same ? fallback when the report is missing, same behaviour under fail_on_broken.

Testing

Rendered the book locally (quarto render)
Ran tests (pytest tests/)
Ran tito module test NN for affected module(s)
Manual verification (describe below)

Pulled the archived lychee report out of the job logs for all nine sweeps in run 31995703306 and replayed both the old and new parser over them. The new count equals lychee's own Errors + Timeouts tally on all nine; the old one matched on four. Timeouts are counted because lychee tallies them separately in the summary table but lists them in the same "Errors per input" section, and a timeout is a link worth triaging.

Nothing outside this repo is needed to reproduce it: the reports are in the logs of that run.

Related Issues

Related to #1810 (the counts and the missing URL lists in that tracker come from this step).

The summary step scanned the whole lychee report for lines shaped `* [<status>] <url>`, which is the wrong set on both ends. Checked against last night's sweep (run 31995703306), the count was wrong on five of the nine sites. Over-counting: the report also has a "Redirects per input" section whose entries have that identical shape. Every followed redirect was counted as a broken link. TinyTorch reported 27 broken when lychee found 1, and 26 of those were `[200]` redirect entries. Four were `[403]`, a status this workflow passes `--accept 200,403` for, so links lychee was explicitly told to treat as fine were still reported broken. Under-counting: the status is not always numeric. A request that never produced an HTTP response is reported as `[ERROR]` or `[TIMEOUT]`, which covers TLS failures, DNS failures, timeouts and missing local files. The `[[:digit:]]+` class could not match those, so they were dropped from both the count and the triage sample. When all of a site's failures were that kind the count parsed to 0 and the "broken but unknown" guard rewrote it to `?`, which is why Book and Slides show `?` in the tracker issue and their URLs are never listed. Book was hiding two TLS failures and Slides a missing slide file plus a timeout. Replaces the two whole-file greps with one awk pass scoped to the "Errors per input" section that accepts any bracketed status. Also strips the markdown autolink delimiters, so the tracker lists `https://example.com` instead of `<https://example.com>`. Verified against the archived reports from all nine sweeps in run 31995703306. The new count now equals lychee's own errors + timeouts tally on every site: Book 2, Labs 0, Kits 0, MLSys-im 0, Slides 2, Instructors 35, TinyTorch 1, Site 1, StaffML 0.

Shashank-Tripathi-07 · 2026-08-20T04:30:23Z

Hi @JOhnsonKC201 !

I Found two correctness bugs in the awk logic, verified against lychee's actual markdown formatter output (checked against lychee-bin/src/formatters/stats/markdown.rs at v0.24.2, the version this workflow's pinned action uses).

1. URL extraction leaves trailing junk (line 196)

The regex only strips a trailing > when it's at the very end of the line. But lychee emits <URL> (at line:col) | reason whenever a link has a span, which is the normal case for markdown/.qmd files, basically everything this workflow scans.

Reproduced with a line copied straight from lychee's own test fixtures:

* [404] <https://github.com/mre/idiomatic-rust-doesnt-exist-man> (at 1:1) | Not Found

After the current awk transforms, BROKEN_SAMPLE ends up with:

https://github.com/mre/idiomatic-rust-doesnt-exist-man> (at 1:1)

instead of a clean URL. This corrupts every entry in the - $url triage list in both the job summary and the nightly issue body.

Fix: strip the (at ...) span suffix (and the trailing >) before using the URL, not just a bare trailing >.

2. Timeouts are never actually counted (line 195)

The awk stops collecting at the next ## heading, apparently intending to bound the "Errors per input" section. But lychee writes ## Timeouts per input as a sibling top-level section, not nested inside "Errors per input" (the formatter emits Errors, then Timeouts, then Ignored, then Redirects, each its own ## section). So the moment ## Timeouts per input appears, collection turns off before any TIMEOUT line is ever read.

That contradicts what the PR is trying to fix: a report whose only failures are timeouts (like the larissasuzuki.com TIMEOUT case cited in the description) produces BROKEN_COUNT = 0, which then gets silently rewritten to ? by the existing guard on line 214. That reproduces the same under-counting/silent-drop failure this PR is meant to fix.

Fix: the section-scoping needs to treat "Errors per input" and "Timeouts per input" as two sections to collect from, instead of stopping at the first ## after Errors begins.

Given #2, I'd also double check the "matches lychee's Errors + Timeouts tally" claim in the manual testing notes. As written, the script can't produce that total for any report where timeouts are present.

please fix these suggested changes and feel free to ping me later on to review those changes too !

@Shashank-Tripathi-07

…span Addresses both bugs @Shashank-Tripathi-07 found in review. Both are real and both are reproducible against lychee's own test fixtures. Timeouts were never counted. lychee writes each outcome as its own top-level section, so "## Timeouts per input" is a sibling of "## Errors per input" and not nested inside it. Bounding the scan at the first "## " after Errors turned collection off the moment the Timeouts heading appeared, so a report whose only failures are timeouts produced a count of 0, which the guard below then rewrote to "?". That is the same silent under-count this step was added to fix. The scan now starts on either heading and still stops at Redirects, Ignored and Suggestions, none of which are failures. URL extraction left trailing junk. lychee appends the source position as "(at line:col)" whenever a link carries a span, so the URL is not last on the line and stripping only a trailing ">" left entries like "https://example.com/foo> (at 1:1)" in the triage list. The span is now removed before the autolink delimiters. Verified two ways. Against a fixture in the current report shape, taken from lychee's own formatter tests, the old parser emitted one corrupted URL and dropped the timeout entirely, while the new one emits both URLs clean and still ignores the redirect and ignored entries. Replaying the nine archived reports from nightly run 31995703306 shows no regression: every site still equals lychee's own errors plus timeouts tally. Worth recording for whoever touches this next: lychee-action v2.8.0 currently defaults to lycheeVersion v0.23.0, whose output has no spans and lists timeouts inside "Errors per input", which is why the nightly tracker has been correct so far. The parser was right for exactly one pinned binary, and a version bump would have broken it quietly.

JOhnsonKC201 · 2026-08-21T03:50:56Z

Thanks @Shashank-Tripathi-07, both of these are real and both are fixed in 52f6666. Reading the formatter rather than just my own output was the right call, and it caught something I would not have.

Timeouts

Confirmed. write_stats_per_input(f, "Timeouts", ...) emits ## Timeouts per input through the same ## {name} per input template as Errors, so it is a sibling and my scan turned off the moment it appeared. The formatter's own test fixture shows the layout plainly, with Errors, Timeouts, Redirects, Ignored and Suggestions all at the same level.

The scan now starts on either heading and still stops at the three that are not failures:

/^##[[:space:]]+(Errors|Timeouts)[[:space:]]+per input[[:space:]]*$/ { collect = 1; next }
/^##[[:space:]]/                                                     { collect = 0 }

The span suffix

Confirmed too. formatted = format!("{formatted} (at {span})") puts the position after the URL, so the URL is not last on the line and my trailing > strip left exactly the corruption you quoted. Removing the span before the autolink delimiters fixes it.

On the tally claim

You were right to flag it, and it is worth pinning down precisely because it decides how urgent this is.

lycheeverse/lychee-action@v2.8.0 defaults to lycheeVersion: v0.23.0, and on that version the output has no spans and lists [TIMEOUT] entries inside ## Errors per input. That is why the nightly tracker has been producing correct numbers so far. Concretely, in the nine archived reports from run 31995703306 the only headings present anywhere are ## Errors per input and ## Redirects per input, no report contains an (at span, and the Slides report has its [TIMEOUT] line sitting under the Errors heading:

## Errors per input

### Errors in ./slides/tinyml/README-edx-original.md

* [ERROR] <file:///.../chapter-3-applications-of-tinyml> | Cannot find file: ...
* [TIMEOUT] <https://larissasuzuki.com/> | Timeout

So the errors-plus-timeouts claim held for the pinned binary. It just held by accident. Being correct against exactly one version is fragile, and bumping lycheeVersion or the action would have broken it silently, which is precisely the failure mode this PR is about. Your fix is the right one regardless of which version is pinned today, so I have made it rather than leaving it as a version note.

Verification

Two ways.

Against a fixture in the current report shape, built from the formatter's own tests:

	URLs emitted
before	`https://github.com/mre/idiomatic-rust-doesnt-exist-man> (at 1:1)`
after	`https://github.com/mre/idiomatic-rust-doesnt-exist-man` `https://httpbin.org/delay/2`

One corrupted URL and a dropped timeout, against two clean URLs, with the Redirects and Ignored entries still correctly excluded.

Then a regression pass replaying all nine archived reports from run 31995703306. No change: every site still equals lychee's own errors plus timeouts tally.

Site	errors + timeouts	before	after
Book	2	2	2
Labs	0	0	0
Kits	0	0	0
MLSys·im	0	0	0
Slides	2	2	2
Instructors	35	35	35
TinyTorch	1	1	1
Unified Site	1	1	1
StaffML	0	0	0

I also updated the comment block above the parser to list all five sections and say which are failures, so the next person does not have to rediscover that Timeouts is a sibling.

Ready for another look whenever you have time.

Shashank-Tripathi-07

Both bugs from my earlier review are fixed correctly.

Timeouts are collected now: Errors and Timeouts are both treated as start-of-collection headings, since they're sibling sections in lychee's output, not nested. A report where every failure is a timeout no longer parses to 0 and gets rewritten to ? by the guard below.

URL extraction is clean: the (at line:col) span is stripped before the autolink delimiters, so entries that carry a source position no longer end up with > (at 12:3) left on the end.

I tested this against a fixture covering all five section types (Errors, Timeouts, Redirects, Ignored, Suggestions), mixing entries with and without a source-position span, a local file:// entry, and a redirect that's a status this workflow accepts. The parser collected exactly the Errors and Timeouts entries, 5 of 5, with clean URLs, and correctly skipped everything else. I also ran a Timeouts-only fixture with no Errors section in the file at all, which is the exact case the old code under-counted, and it collected correctly.

Good call flagging in the commit message that this only works for the currently pinned lychee version's output shape. Worth a comment in the workflow or a version pin note as a follow-up, but not a blocker for this PR.

Approving.

Land four more contributor PRs, each audited by running it rather than reading it, with the defects found repaired here: #2015 bare 'tito package reset' now actually resets #2023 conftest validates all 20 module exports #2080 link-check counts from the failure sections only #2092 Jupyter server reuse + start/resume desync recovery Three of the four had a real defect that only surfaced under execution: #2015 --force before the SUBCOMMAND token was silently dropped by the subparser default, then blocked on input() (EOFError in CI) #2023 the module-20 registry path never resolved, so every pytest run on a fully-exported tree printed a false 'not exported' warning, and two of the PR's own tests asserted the wrong value #2092 the pid check returned True for any live process on macOS, so a recycled pid would permanently block launching Jupyter #2080 needed no repair: its three claims check out against real lychee 0.23.0 output.

profvjreddi · 2026-09-01T11:48:48Z

Thanks @JOhnsonKC201. Integrated into dev unchanged, CI green.

I went and checked this against real lychee output rather than taking the reasoning on faith, and all three of your points hold. The section headings are exactly ## Errors per input and ## Redirects per input, they are siblings, and ### Errors in <file> correctly doesn't close the scan since ^##[[:space:]] needs whitespace after the hashes.

The old expression was worse than the PR describes. On a report with one [ERROR] and one followed redirect it returned 1, but by counting the redirect and missing the error:

truth: 1 error, 1 followed redirect
OLD code (whole-file, digits only) : 1   ← the [200] redirect
this PR                            : 1   ← the [ERROR]

Same number, wrong item, which is exactly the kind of bug that survives review. Add a second redirect and the old count is simply wrong.

Nothing needed changing. Thanks for the thorough comments in the awk too, they made this quick to verify.

@all-contributors please add @JOhnsonKC201 for code, tool

github-actions · 2026-09-01T11:51:03Z

I couldn't determine which project(s) to add the contributor to. 🤔

Your comment: @all-contributors please add @JOhnsonKC201 for code, tool

This repo has multiple projects. Specify one or more explicitly, e.g.:

@all-contributors @JOhnsonKC201 for code, tool in tinytorch
@all-contributors @JOhnsonKC201 for code, tool in TinyTorch, Book, Kits

How project detection works:

In comment: Say "in TinyTorch", "for book, labs", etc. (multiple projects OK)
On PRs: Auto-detected from changed file paths when only one project is touched
On issues: From labels or title, or specify in the comment

profvjreddi · 2026-09-01T11:53:03Z

Filing this under book: infra-link-check.yml is shared by eleven validate workflows, so it has no project directory of its own, and its default lycheeignore_path lives under book/.

@all-contributors please add @JOhnsonKC201 for code, tool in book

github-actions · 2026-09-01T11:55:55Z

I've added @JOhnsonKC201 as a contributor to book! 🎉

Recognized for: code, tool
Project(s): book (explicitly mentioned in comment)
Based on: @all-contributors please add @JOhnsonKC201 for code, tool in book

The contributor list has been updated in:

book/.all-contributorsrc, book/README.md
Main README.md

We love recognizing our contributors! ❤️

JOhnsonKC201 · 2026-09-01T15:00:58Z

Thanks @profvjreddi, and for checking it against real output rather than the write-up. The one-error-one-redirect case is sharper than anything I had: same count, wrong item, which is exactly what a diff-only review misses.

profvjreddi · 2026-09-01T16:28:38Z

No worries. More importantly, I appreciate you taking the time to issue a PR and contribute. Absolutely. This is one of my favorite things about doing all this stuff: I get to interact with so many of you through the sort of real work.

github-actions Bot added area: tools Build tools, scripts, CI/CD link-health type: bug bug in rendering labels Aug 18, 2026

Shashank-Tripathi-07 assigned JOhnsonKC201 Aug 20, 2026

Shashank-Tripathi-07 approved these changes Aug 26, 2026

View reviewed changes

Shashank-Tripathi-07 requested a review from profvjreddi August 26, 2026 11:44

profvjreddi closed this in 793e24d Aug 31, 2026

Navigation Menu

Uh oh!

fix(link-check): count broken links from the errors section only - #2080

fix(link-check): count broken links from the errors section only#2080
JOhnsonKC201 wants to merge 2 commits into
harvard-edge:devfrom
JOhnsonKC201:fix/link-check-count-errors-section

JOhnsonKC201 commented Aug 17, 2026

Uh oh!

Shashank-Tripathi-07 commented Aug 20, 2026 •

edited

Loading

Uh oh!

JOhnsonKC201 commented Aug 21, 2026

Uh oh!

Shashank-Tripathi-07 left a comment

Uh oh!

profvjreddi commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

profvjreddi commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

JOhnsonKC201 commented Sep 1, 2026

Uh oh!

profvjreddi commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

JOhnsonKC201 commented Aug 17, 2026

Summary

Area

Changes

Testing

Related Issues

Uh oh!

Shashank-Tripathi-07 commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. URL extraction leaves trailing junk (line 196)

2. Timeouts are never actually counted (line 195)

Uh oh!

JOhnsonKC201 commented Aug 21, 2026

Timeouts

The span suffix

On the tally claim

Verification

Uh oh!

Shashank-Tripathi-07 left a comment

Choose a reason for hiding this comment

Uh oh!

profvjreddi commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

profvjreddi commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

JOhnsonKC201 commented Sep 1, 2026

Uh oh!

profvjreddi commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Shashank-Tripathi-07 commented Aug 20, 2026 •

edited

Loading