Skip to content

test(server): harden and evaluate the existing security mutation suite #1498

Description

@FelixTJDietrich

Hephaestus already has a PIT mutation profile, introduced in #1317, for six security-sensitive server classes. The immediate task is to make that existing suite trustworthy and decide whether its value justifies ongoing ownership—not to add Stryker or a second mutation framework.

A local run on 2026-08-26 produced:

Status Count
Generated mutants 203
Killed 137
Survived 18
No coverage 40
Run error 8

PIT analysis took about 1m29s and the complete Maven command about 3m01s. All eight run errors were in EncryptionContext, yet Maven returned BUILD SUCCESS. Survivors affected behavior in ServerUrlValidator, ReturnToValidator, AdminBootstrapPolicy and IssuerDiscoveryProbe.

That silent-success mode is more urgent than a mutation-score threshold.

Scope

  • Reproduce after perf(server): reuse generated client build outputs #1529's Maven layout lands.
  • Eliminate RUN_ERROR, TIMED_OUT and MEMORY_ERROR, and fail the wrapper/workflow when PIT reports a technical analysis failure even if Maven exits zero.
  • Triage the current 18 survivors and 40 no-coverage mutations.
  • Add tests where a mutant exposes genuinely unverified behavior.
  • Document narrowly justified equivalent or unproductive mutants instead of adding implementation-coupled tests merely to improve a percentage.
  • Expose a reproducible manual, non-required workflow with an actionable summary and HTML/XML artifacts.

Do not use PIT's experimental incremental analysis for this initial 203-mutant scope. Do not promote mutation testing to required PR CI in this issue.

Evaluation contract

Name an owner for mutant triage and agree a runner-time budget before starting. Record:

  • setup/compilation time;
  • mutation-analysis time;
  • human triage time;
  • actionable survivors;
  • equivalent/unproductive mutants;
  • tests improved.

Done when

  • Technical analysis errors cannot report a successful mutation run.
  • Existing survivors and uncovered mutations are triaged.
  • Reports and summaries are reproducible from a manual non-required workflow.
  • A decision is recorded: drop, keep manual/advisory, run periodically, or open a separate required-gate proposal.
  • This evaluation issue is closed after that decision.

Mutation testing cannot detect omitted implementations, prevent test/config tampering, or replace integration, E2E and independent review. A score is not the goal; useful fault detection at acceptable compute and attention cost is.

References:

Metadata

Metadata

Assignees

No one assigned

    Labels

    evaluationEvaluation, audit, or research-enabling product infrastructuresize:MThis PR changes 30-99 lines, ignoring generated files.status:blockedBlocked by external dependency or other issuetestUnit tests, integration tests, test infrastructure

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions