Summary
validate-doc-claims.py reports any bare 7-40 character hex string as a fabricated commit SHA. Session identifiers, content hashes and blob hashes all match, so a doc that legitimately cites one gets a flag that reads as a defect.
Cause
SHA_RE matches any hex word:
SHA_RE = re.compile(r"\b[0-9a-f]{7,40}\b")There is a guard immediately after it, but it filters the wrong thing: it requires the candidate to contain both a digit and a letter, which excludes deadbeef while admitting every realistic session id.
if not (any(c.isdigit() for c in sha) and any(c in "abcdef" for c in sha)):
continue # dates and decimal ids are not SHAsReproduction
A doc containing a transcript excerpt that names its sessions:
session 7e6861b4: Write -> /tmp/attribute.sh
session dc828513: Write -> docs/solutions/conventions/some-doc.md
produces:
FLAG sha 7e6861b4 (line 220) - does not resolve to a commit in this repository. Replace with the PR number, or drop it.
FLAG sha dc828513 (line 222) - does not resolve to a commit in this repository. Replace with the PR number, or drop it.
Neither is a commit reference, and the suggested remedy ("replace with the PR number") does not apply to a session id.
Observed on compound-engineering 3.23.4. Three such flags in one doc during a refresh run, all adjudicated as false positives.
Why it matters beyond the noise
The docstring is explicit that flags are adjudication input rather than hard failures, and that is the right design. But the value of an adjudicated flag depends on flags being worth reading. A validator that reliably flags a legitimate citation format trains the reader to skim past its output, at which point a real fabricated SHA passes with the rest.
This is the same class as #1212 (legitimate {{PLACEHOLDER}} content flagged as scaffold), which was closed as completed, and it is on the same script as #1545.
Possible directions
Not a PR, since which of these fits the project's intent is the maintainers' call:
- Require a citation-shaped context before flagging, rather than a bare hex word: a leading
commit, sha, #, or a git-ish verb nearby. Highest precision, some recall cost.
- Skip a candidate immediately preceded by a word that denotes something else,
session being the common one.
- Keep detection as-is but soften the flag text for a hex string with no citation context, so the reader is told the script cannot tell what kind of identifier it is looking at, rather than that a commit does not exist.
Happy to test whichever direction you prefer against a corpus that currently trips it.
Summary
validate-doc-claims.pyreports any bare 7-40 character hex string as a fabricated commit SHA. Session identifiers, content hashes and blob hashes all match, so a doc that legitimately cites one gets a flag that reads as a defect.Cause
SHA_REmatches any hex word:There is a guard immediately after it, but it filters the wrong thing: it requires the candidate to contain both a digit and a letter, which excludes
deadbeefwhile admitting every realistic session id.Reproduction
A doc containing a transcript excerpt that names its sessions:
produces:
Neither is a commit reference, and the suggested remedy ("replace with the PR number") does not apply to a session id.
Observed on compound-engineering 3.23.4. Three such flags in one doc during a refresh run, all adjudicated as false positives.
Why it matters beyond the noise
The docstring is explicit that flags are adjudication input rather than hard failures, and that is the right design. But the value of an adjudicated flag depends on flags being worth reading. A validator that reliably flags a legitimate citation format trains the reader to skim past its output, at which point a real fabricated SHA passes with the rest.
This is the same class as #1212 (legitimate
{{PLACEHOLDER}}content flagged as scaffold), which was closed as completed, and it is on the same script as #1545.Possible directions
Not a PR, since which of these fits the project's intent is the maintainers' call:
commit,sha,#, or a git-ish verb nearby. Highest precision, some recall cost.sessionbeing the common one.Happy to test whichever direction you prefer against a corpus that currently trips it.