Skip to content

PR_MUTATION_TESTING.md

Latest commit

 

History

History
135 lines (98 loc) · 6.44 KB

PR_MUTATION_TESTING.md

File metadata and controls

135 lines (98 loc) · 6.44 KB
Raw

Description

Adds starforge mutate, an AI mutation-testing suite that measures how effective a contract's tests actually are. It introduces small faults ("mutants") into Soroban contract source, runs the test suite against each one, and reports which mutations slipped through undetected — each survivor is a proven blind spot in the suite.

Unlike line coverage, which only shows what code ran, mutation testing shows what the tests actually verify.

Closes #568

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Changes Made

  • Add src/utils/mutation.rs — the mutation engine: 10 mutation operators, mutant generation, scoring, per-operator/per-function analysis, weak-spot detection, test-improvement suggestions, and markdown/text/HTML reporting.
  • Add src/commands/mutate.rs — the starforge mutate CLI (generate, run, operators, ci-workflow) plus the real subprocess test executor with timeout handling and guaranteed source restoration.
  • Register the modules and wire the Mutate command into src/main.rs, src/commands/mod.rs, and src/utils/mod.rs.
  • Add tests/mutation_testing.rs — CLI integration tests.
  • Incidental build fix (same as #508, since this branch is cut from master): remove stray match-arm lines inside enum Commands, wire the unregistered Migrate variant, drop a duplicate pub mod simulate;.

Mutation operators

arithmetic, comparison, boolean, logical, assignment, constant, negation, plus three Soroban-specific ones: require-auth (deletes an authorisation check), storage-durability (swaps persistent/instance/temporary), and unwrap-default.

Requirements coverage

Requirement Implementation
Mutation generation 10 operators; string/comment/#[cfg(test)]-aware so mutants compile
Test execution Subprocess runner, per-mutant timeout, baseline verification
Mutation analysis Score excludes non-compiling mutants; timeouts count as detected
Coverage reporting text / markdown / HTML / JSON, per-operator and per-function
Weak test detection Functions ranked by surviving mutants
Improvement suggestions Severity-ranked, operator-specific recommendations
Performance optimization Deterministic stride sampling (--max-mutants), operator filtering, dedup, timeouts
CI/CD integration --min-score + --ci gating; mutate ci-workflow emits a GitHub Actions job

Testing

How has this been tested?

The full crate could not be compiled in the dev sandbox (offline registry — see Additional Context), so the logic was validated by compiling the relevant modules standalone with rustc:

  • Engine: 25 unit tests pass — generation per operator, string/comment/test-module exclusion, function attribution, dedup, deterministic sampling, scoring maths, weak spots, suggestions, rendering, CI YAML.
  • Executor harness: 6 scenarios pass — exit-0 → survived, exit-1 → killed, compile error → build-failed, timeout fires promptly, SourceGuard restores the file, and a >64 KB output run completes in ~60 ms without deadlocking.
  • Generation sanity check on a realistic Vault contract produced 16 meaningful mutants across 9 operators with correct function attribution, and every invariant held (each mutant changes exactly one line; no mutants leak from the test module).

Reproduce:

cargo test --lib mutation
cargo test --test mutation_testing
  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed

Test Coverage

Describe what scenarios have been tested:

  • Happy path: generate mutants, run with a failing suite (all killed, score 100%), run with a passing suite (all survived, suggestions + weak spots emitted), markdown report written to disk, CI workflow generated.
  • Edge cases: operator filtering, --max-mutants capping with deterministic sampling, generics (Vec<u32>) not mistaken for comparisons, mutations inside strings/comments/test modules suppressed, >64 KB test output, per-mutant timeout.
  • Error handling: missing source file, unknown operator slug, red baseline aborts the run, CI gate exits non-zero below --min-score, and the contract source is always restored (even on panic, via a Drop guard).

Code Quality Checklist

  • My code follows the style guidelines of this project (cargo fmt)
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings (cargo clippy -- -D warnings)
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • The CI checks pass (format, clippy, tests)

Breaking Changes

  • This PR introduces breaking changes

None — this adds a new, self-contained command and does not alter existing behaviour.

Documentation

  • README.md updated
  • DEVELOPER_GUIDE.md updated (if applicable)
  • API_REFERENCE.md updated (if applicable)
  • No documentation changes needed

Module- and command-level rustdoc is included; usage is discoverable via starforge mutate --help and starforge mutate operators.

Screenshots (if applicable)

N/A — CLI feature.

Additional Context

Safety. mutate run rewrites the target source file in place while testing each mutant. A Drop-based SourceGuard restores the original contents on every exit path, including panics and early bail!s; an integration test asserts the file is byte-identical after a run.

Known limitation. On timeout only the spawned shell is killed, not its descendants, so a runaway cargo test may linger until it finishes. The --timeout default (120 s) is deliberately generous. Documented in-code.

Build environment. The crates.io registry was unreachable and tokio's transitive dependencies were not cached, so cargo build could not run. The engine (pure std + serde derives) and the executor were validated standalone with rustc; the CLI layer follows existing command patterns (test.rs, lint.rs, migrate.rs). Please run the checks below in CI.

Note: Make sure all tests pass locally before submitting:

cargo test
cargo fmt --all
cargo clippy -- -D warnings