Outcome
Hephaestus does not spend review budget on work no person did, and does not comment on files a repository
generates. What it ignores is stated per repository, and most of it is proposed automatically from what
the repository already declares to its other tools.
Problem
Measured across three real repositories (ls1intum/Artemis, ls1intum/Hephaestus, prompt-edu/prompt),
over the last 300 first-parent commits of each:
|
Artemis |
Hephaestus |
prompt |
| Pull requests authored by bots |
0% |
17.4% |
39.8% |
| Changed lines that are generated |
15.9% |
28.2% |
36.1% |
| Pull requests touching generated paths |
24.7% |
46.5% |
52.8% |
Applying the shipped catalogue to those repositories gives 111 practice × repository cells, of which
25 are noisy — the practice fires, and the finding is not useful. The two worst are structural:
excludes-generated-and-build-artifacts fires on 46.5% of Hephaestus pull requests and is wrong every
time, because committing the generated client is deliberate, documented policy and CI pushes it.
changes-dependencies-deliberately fires on 58% of prompt pull requests; two thirds of those are
Renovate and Dependabot.
There is also a reviewer-side case: Artemis runs an AI reviewer with request_changes_workflow: true, so
the three practices with subject: REVIEWER would grade a bot.
None of this is expressible today.
What
- Refuse bot work, on both sides. Bot authors, and bot reviewers — the reviewer case is not covered by
an author rule, and a typed refusal reason must say which applied.
- Per-repository ignore rules for paths. Per repository because the namespace collides:
docs/**
means three different things across these three repositories, and one global list cannot serve them.
- Decide and document the semantics. "Ignore this path" has three defensible readings — remove the
files from the diff, drop the practice for this change, or suppress the resulting observation — and they
produce different reviews. At least one shipped practice exists to look at generated files, so a
diff-stripping rule would blind it. Pick one, write it down, and make the trace say which applied.
- Propose the rules from what the repository already declares.
renovate.json ignorePaths,
.gitattributes linguist-generated, Code generated … DO NOT EDIT headers, .pre-commit-config.yaml
exclusions. Roughly half of the setup work for these three repositories is transcribing facts the
repository already states. This reads other tools' declarations only, grants no privilege, and is a
proposal an admin confirms — it is not a Hephaestus rules file.
Acceptance criteria
Out of scope
- Reading practice definitions, rules or instructions from the reviewed repository. Only ignore
declarations already written for other tools are read, and only as a proposal.
- Fixing the practice criteria that are noisy for population reasons rather than path reasons — for
example, the commit-hygiene practices on a repository that squash-merges everything. Those need a
catalogue decision, not an ignore rule.
Outcome
Hephaestus does not spend review budget on work no person did, and does not comment on files a repository
generates. What it ignores is stated per repository, and most of it is proposed automatically from what
the repository already declares to its other tools.
Problem
Measured across three real repositories (
ls1intum/Artemis,ls1intum/Hephaestus,prompt-edu/prompt),over the last 300 first-parent commits of each:
Applying the shipped catalogue to those repositories gives 111 practice × repository cells, of which
25 are noisy — the practice fires, and the finding is not useful. The two worst are structural:
excludes-generated-and-build-artifactsfires on 46.5% of Hephaestus pull requests and is wrong everytime, because committing the generated client is deliberate, documented policy and CI pushes it.
changes-dependencies-deliberatelyfires on 58% of prompt pull requests; two thirds of those areRenovate and Dependabot.
There is also a reviewer-side case: Artemis runs an AI reviewer with
request_changes_workflow: true, sothe three practices with
subject: REVIEWERwould grade a bot.None of this is expressible today.
What
an author rule, and a typed refusal reason must say which applied.
docs/**means three different things across these three repositories, and one global list cannot serve them.
files from the diff, drop the practice for this change, or suppress the resulting observation — and they
produce different reviews. At least one shipped practice exists to look at generated files, so a
diff-stripping rule would blind it. Pick one, write it down, and make the trace say which applied.
renovate.jsonignorePaths,.gitattributeslinguist-generated,Code generated … DO NOT EDITheaders,.pre-commit-config.yamlexclusions. Roughly half of the setup work for these three repositories is transcribing facts the
repository already states. This reads other tools' declarations only, grants no privilege, and is a
proposal an admin confirms — it is not a Hephaestus rules file.
Acceptance criteria
none of them without confirmation.
named with the reason they cannot be fixed by an ignore rule.
Out of scope
declarations already written for other tools are read, and only as a proposal.
example, the commit-hygiene practices on a repository that squash-merges everything. Those need a
catalogue decision, not an ignore rule.