Skip to content

feat(server,webapp): exclude bot work and generated paths from practice review #1508

Description

@FelixTJDietrich

Outcome

Hephaestus does not spend review budget on work no person did, and does not comment on files a repository
generates. What it ignores is stated per repository, and most of it is proposed automatically from what
the repository already declares to its other tools.

Problem

Measured across three real repositories (ls1intum/Artemis, ls1intum/Hephaestus, prompt-edu/prompt),
over the last 300 first-parent commits of each:

Artemis Hephaestus prompt
Pull requests authored by bots 0% 17.4% 39.8%
Changed lines that are generated 15.9% 28.2% 36.1%
Pull requests touching generated paths 24.7% 46.5% 52.8%

Applying the shipped catalogue to those repositories gives 111 practice × repository cells, of which
25 are noisy — the practice fires, and the finding is not useful. The two worst are structural:

  • excludes-generated-and-build-artifacts fires on 46.5% of Hephaestus pull requests and is wrong every
    time, because committing the generated client is deliberate, documented policy and CI pushes it.
  • changes-dependencies-deliberately fires on 58% of prompt pull requests; two thirds of those are
    Renovate and Dependabot.

There is also a reviewer-side case: Artemis runs an AI reviewer with request_changes_workflow: true, so
the three practices with subject: REVIEWER would grade a bot.

None of this is expressible today.

What

  • Refuse bot work, on both sides. Bot authors, and bot reviewers — the reviewer case is not covered by
    an author rule, and a typed refusal reason must say which applied.
  • Per-repository ignore rules for paths. Per repository because the namespace collides: docs/**
    means three different things across these three repositories, and one global list cannot serve them.
  • Decide and document the semantics. "Ignore this path" has three defensible readings — remove the
    files from the diff, drop the practice for this change, or suppress the resulting observation — and they
    produce different reviews. At least one shipped practice exists to look at generated files, so a
    diff-stripping rule would blind it. Pick one, write it down, and make the trace say which applied.
  • Propose the rules from what the repository already declares. renovate.json ignorePaths,
    .gitattributes linguist-generated, Code generated … DO NOT EDIT headers, .pre-commit-config.yaml
    exclusions. Roughly half of the setup work for these three repositories is transcribing facts the
    repository already states. This reads other tools' declarations only, grants no privilege, and is a
    proposal an admin confirms — it is not a Hephaestus rules file.

Acceptance criteria

  • A pull request authored by a bot produces no review work, with a typed reason in the trace.
  • A review that would grade a bot reviewer does not, with its own typed reason.
  • Ignore rules are set per repository and their semantics are documented and visible in the trace.
  • Connecting a repository offers the ignore rules its own configuration already implies, and applies
    none of them without confirmation.
  • On the three repositories above, the measured noisy-cell count falls, and the remaining cells are
    named with the reason they cannot be fixed by an ignore rule.

Out of scope

  • Reading practice definitions, rules or instructions from the reviewed repository. Only ignore
    declarations already written for other tools are read, and only as a proposal.
  • Fixing the practice criteria that are noisy for population reasons rather than path reasons — for
    example, the commit-hygiene practices on a repository that squash-merges everything. Those need a
    catalogue decision, not an ignore rule.

Metadata

Metadata

Assignees

No one assigned

    Labels

    application-serverSpring Boot server: APIs, business logic, databasefeatureNew feature or enhancementpriority:highAddress this sprint - Significant impactwebappReact app: UI components, routes, state management

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions