Decouple the JQF/Zest engine from JUnit 4 and junit-quickcheck (#80) by vlsi · Pull Request #278 · rohanpadhye/JQF

vlsi · 2026-07-01T06:03:27Z

Why

The JQF/Zest engine was hard-wired to JUnit 4 and junit-quickcheck, both end-of-life. To describe a fuzz target, test code had to pull in the whole engine and both legacy libraries, and the engine could not be driven from any other test framework. Issue #80 asks to split the API from the engine; this PR carries that through to a full decoupling.

What

jqf-core is now an independent engine with no compile or runtime dependency on a test framework or on junit-quickcheck. Argument generation and trial execution moved behind an SPI in edu.berkeley.cs.jqf.fuzz.spi, so the two axes — how to generate arguments and how to run a trial — are pluggable and independent.

The existing way to run tests is preserved: @RunWith(JQF.class) + @Fuzz, ZestGuidance, and GuidedFuzzing still work through the dependency-only jqf-fuzz aggregator, and junit-quickcheck becomes one generator provider (jqf-generator-quickcheck, with junit:junit excluded).

New capabilities:

jqf-junit5 — a JUnit 5 run path via @FuzzTest (@TestTemplate + a Jupiter extension run by the stock junit-jupiter-engine), free of JUnit 4. A plain mvn test replays the saved corpus, seeds, and one empty input as a bounded regression; -Djqf.fuzz=true (or JQF_FUZZ, repro, or an installed guidance) runs a full Zest campaign and fails the node on the first finding.
maven-plugin — jqf:fuzz and jqf:repro now fuzz and replay a JUnit 5 @FuzzTest as well as a JUnit 4 @Fuzz. A FuzzTestDispatcher picks the run path from the method's annotations and normalises the JUnit 4 Result and the engine's FuzzResult; the JUnit 4 path is unchanged.
Two more generator providers prove the SPI is generation-library agnostic: jqf-generator-instancio (seed-based) and jqf-generator-jetcheck (byte-structured — jetCheck reads randomness as a stream of ints, so the fuzzer's byte mutations map onto local structural changes in the value, a better fit for coverage-guided search).

Module layout

jqf-instrument            ASM/janala, unchanged
jqf-core                  engine + SPI, no junit, no junit-quickcheck
jqf-generator-quickcheck  reference generator provider (junit:junit excluded)
jqf-generator-instancio   seed-based generator provider (Instancio)
jqf-generator-jetcheck    byte-structured generator provider (jetCheck)
jqf-junit4                @RunWith(JQF.class), GuidedFuzzing, TrialRunner
jqf-junit5                @FuzzTest, Jupiter extension
jqf-fuzz                  dependency-only aggregator, keeps existing coordinates
maven-plugin              depends on jqf-core + jqf-junit4 + jqf-junit5

Commits

The branch is seven focused commits, reviewable in order:

refactor(fuzz): extract junit-free FuzzRunner and pluggable SPI — the SPI, FuzzResult, the extracted loop, Guidance.run(...) replaced by beforeRun() + injected SPI, DiffFuzz via a DiffTrialExecutor.
refactor(fuzz): add junit-free engine entry and move CLI off GuidedFuzzing — FuzzRunner.run(...) returning FuzzResult; CLI and drivers migrated.
refactor: split jqf-fuzz into core, generator, and junit4 modules — the physical module split.
feat(junit5): add JUnit 5 (Jupiter) adapter for running JQF fuzz tests — jqf-junit5.
feat(maven-plugin): fuzz and replay JUnit 5 @FuzzTest methods — the plugin dispatcher.
feat(generator): add Instancio argument-generator provider.
feat(generator): add jetCheck argument-generator provider.

Backwards compatibility

@RunWith(JQF.class) + @Fuzz, ZestGuidance, and GuidedFuzzing are unchanged and reachable through jqf-fuzz; the edu.berkeley.cs.jqf.fuzz package is split across jars on purpose (Fuzz/FuzzRunner in jqf-core, JQF in jqf-junit4) to keep those imports, which is fine on the classpath (no module-info).
Guidance.run(TestClass, FrameworkMethod, Object[]) was removed in favour of beforeRun() + the SPI. Inside the repo only ExecutionIndexingGuidance and DiffFuzzReproGuidance used it; both are migrated. An external Guidance that overrode run(...) breaks at compile time.
jqf-junit5 is opt-in: it is not pulled in transitively by jqf-fuzz, so existing JUnit 4 users are unaffected.

Upstream follow-ups for the generator providers

jetCheck drives generation through the public GenerationEnvironment.generative(IntSource, int), added in jetCheck 0.3.0 — the entry point requested in JetBrains/jetCheck#7. An earlier revision reached jetCheck's package-private internals; 0.3.0 removed that need, so the provider now depends on jetCheck 0.3.0 with no internal coupling.
Instancio derives all randomness from a single seed, so Zest can only steer it coarsely — a byte change reshapes the whole object rather than one field. org.instancio.Random is public but cannot be injected. Requested upstream: instancio/instancio#1770.

How to verify

mvn -Pcontinuous-integration clean install is green. Notable tests:

ZestGuidanceIT (4) — the JUnit 4 path and the FuzzRunner.run engine entry produce the same campaign (same corpus, same hashes).
JUnit5ExecutorIT (1) — a real instrumented campaign through the JUnit 5 executor explores the same seven branches as the JUnit 4 path.
FuzzTestDispatcherIT (4) — the plugin detects both styles, fuzzes a planted-bug @FuzzTest, replays the saved input, and runs the JUnit 4 path unchanged.
jqf-junit5 (21), jqf-generator-instancio (4), jqf-generator-jetcheck (5) — mode selection, fail-fast without a provider, determinism, and @FuzzTest end-to-end via ServiceLoader.

mvn -pl jqf-core dependency:tree and the two new provider modules show no junit:junit and no com.pholser on the compile/runtime classpath.

🤖 Generated with Claude Code

vlsi · 2026-07-01T06:56:30Z

@rohanpadhye — could you confirm the copyright-header policy for the new files before merge?

They repeat the header copied from existing sources, Copyright (c) 2017-2018 The Regents of the University of California. That year is wrong for files written now, and no header-enforcing plugin (license-maven-plugin, RAT, Checkstyle) runs in the build, so it's purely a metadata choice.

git diff -M splits the change into 60 moved files (headers untouched) and 47 genuinely new ones. For the new files, to match LICENSE (Copyright (c) 2021-2023 Rohan Padhye and JQF Contributors), I propose:

New original files: Copyright (c) 2026 Rohan Padhye and JQF Contributors.
Files derived from existing code (FuzzRunner, the quickcheck ArgumentsGenerator*, the rewritten FuzzStatement): keep the original Regents and Padhye lines and add the 2026 ... and JQF Contributors line.
The 60 moved files: left unchanged.

Let me know the holder and year format you prefer, and I'll apply it in a follow-up commit.

rohanpadhye · 2026-07-01T12:20:44Z

@vlsi

Thanks for the PR. This is a big change. I was actually planning on retiring from JQF maintaining this year and archiving the repository. Are you interested in maintaining a fork that goes beyond JUnit 4? I would be happy to leave a link to your fork from the archival commit.

I don't have an opinion on the copyright headers. Anything that complies with the BSD 2-style-clause is fine. When I started this project in 2018 or so, I was a graduate student researcher at UC Berkeley and so the original copyright officially belonged to the UC Regents. All my contributions after 2020 as well as contributions have not required this notice, and so I started using "Rohan Padhye and JQF Contributors" instead. If someone else takes over, they can use their own copyright header as long as previous ones in each file retain the markers (say on separate lines) as per the license requirement.

vlsi · 2026-07-01T13:26:48Z

Thanks for the quick reply, and for offering to link the fork.

For context: I'm prototyping jqf + jetCheck to test pgjdbc/pgjdbc#3062.

Are you interested in maintaining a fork that goes beyond JUnit 4?

Honestly, I'd rather contribute upstream than maintain a fork. I don't expect many changes to jqf; JUnit 4 is forbidden in pgjdbc, which is why I opened this PR.

If you do archive the repository, I'd keep my fork alive for a while so this stays usable.

vlsi · 2026-07-04T21:34:43Z

@rohanpadhye , I've updated copyrights, and I'm more-or-less happy with the change.

I've made progress with jqf-based tests (see pgjdbc/pgjdbc@706fd8d), so I wonder what is your vision on merge/release jqf.

rohanpadhye · 2026-07-08T21:09:48Z

I need to set aside some time to review this but I can plausibly cut a 3.0 release since this is quite a major change.

Decouple the Zest engine from JUnit 4 and junit-quickcheck (issue rohanpadhye#80). Add an edu.berkeley.cs.jqf.fuzz.spi package (ArgumentsGenerator, TrialExecutor, ResultClassifier, FuzzFramework, SkipTrialException) plus FuzzResult and MultipleFailuresError, and move the fuzzing loop out of FuzzStatement into a framework-independent FuzzRunner. Replace Guidance.run(TestClass, FrameworkMethod, Object[]) with Guidance.beforeRun() and the injected SPI, and classify INVALID trials through ResultClassifier so the engine no longer references org.junit. Differential fuzzing moves to a DiffTrialExecutor decorator and DiffFuzzGuidance.acceptOutcome. The JUnit 4 + junit-quickcheck integration becomes a set of providers (Junit4*, Quickcheck*) registered via ServiceLoader; FuzzStatement is now a thin bridge. Engine packages (guidance, difffuzz, ei guidances, util, random, repro, afl, spi) no longer import org.junit or com.pholser. ZestCLI and ZestDriver still use the JUnit 4 GuidedFuzzing orchestrator and move onto the engine entry with the upcoming module split. Behaviour is unchanged: the fuzz unit tests and ZestGuidanceIT pass, and the @RunWith(JQF.class) examples compile without edits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…zzing Add static FuzzRunner.run(...) overloads: a campaign entry that discovers the ArgumentsGeneratorFactory and FuzzFramework through ServiceLoader, wires the guidance's instrumentation callback, and returns a FuzzResult instead of org.junit.runner.Result. Migrate ZestDriver and ZestCLI onto it, so the engine packages no longer import org.junit (the grep gate is now clean). Clear this thread's tracer at the start of a campaign and register the callback before snooping begins. Without this, a second campaign in the same JVM could reuse a tracer bound to the previous campaign's guidance and collect no coverage. ZestGuidanceIT now also runs PatriciaTrie and SimpleClass through FuzzRunner.run and asserts the same corpus and coverage as the GuidedFuzzing path, confirming the two entry points are equivalent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@fuzz

Issue rohanpadhye#80 asks for the JQF/Zest engine to stand on its own, free of JUnit 4 and junit-quickcheck, with junit-quickcheck demoted to one pluggable generator provider. The previous two commits decoupled the engine in place; this commit performs the physical module split they prepared. New modules: - jqf-core: the test-framework-independent engine (guidance, ei, util, random, repro, afl, difffuzz, spi, and Fuzz/FuzzRunner/FuzzResult/ZestCLI). Its compile and runtime classpath carries no JUnit and no junit-quickcheck. - jqf-generator-quickcheck: the reference ArgumentsGenerator provider. It depends on junit-quickcheck with junit:junit excluded; the generator packages carry no reference to JUnit 4 (verified with jdeps), and JQF never uses the junit-quickcheck runner. - jqf-junit4: the JUnit 4 runner adapter (JQF, GuidedFuzzing, TrialRunner, the Junit4 SPI providers, and FuzzStatement). jqf-fuzz keeps its coordinates and becomes a dependency-only aggregator over the three modules, so existing users of edu.berkeley.cs.jqf:jqf-fuzz still get JQF, @fuzz, ZestGuidance, GuidedFuzzing, and the rest. The zest-cli assembly stays here because it needs every module and SPI provider on one classpath. The six remaining engine drivers (AFL, PerfFuzz, Repro, ReproServer, ExecutionIndexing, Random) now call FuzzRunner.run instead of the JUnit 4 GuidedFuzzing. Notes on a few non-mechanical choices: - FuzzStatement moves from ...junit.quickcheck to ...junit so that package is not split across jars; the ...fuzz package is split on purpose (Fuzz and FuzzRunner in jqf-core, JQF in jqf-junit4) to preserve @fuzz and @RunWith(JQF.class) imports. The project has no module-info, so split packages are fine on the classpath. - GuidanceTest moves to jqf-junit4 rather than jqf-core: it runs through @RunWith(JQF.class) and GuidedFuzzing, so keeping it in jqf-core would create a reactor cycle. The other six unit tests stay in jqf-core and exercise the engine through JUnit 4 and junit-quickcheck at test scope only. - The afl-proxy C source stays under fuzz/src/main/c; the Makefile builds it from that path. Also point the API Docs workflow at the new module layout: aggregate every library module by exclusion (all but jqf-examples and the Maven plugin) instead of the hand-listed jqf-instrument/jqf-fuzz set, so jqf-core and every module added later are documented too. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Phase 4 of issue rohanpadhye#80: give Zest a junit4-free way to run fuzz tests, so the engine no longer has to be driven through JUnit 4. `@FuzzTest` is a `@TestTemplate` plus `@ExtendWith(JQFTestExtension.class)`, run by the standard junit-jupiter-engine rather than a custom TestEngine. The extension has two modes. A plain `mvn test` replays the saved corpus, the seed files, and one empty input as a bounded regression under the normal Jupiter lifecycle; an input whose arguments cannot be generated (a constrained generator that the input cannot satisfy) is skipped rather than run with null. `-Djqf.fuzz=true` (or the `JQF_FUZZ` environment variable, a repro input, or an installed guidance) instead drives a full Zest campaign through FuzzRunner and fails the node on the first finding. The module is generator-agnostic: it depends only on jqf-core and Jupiter, not on jqf-generator-quickcheck or junit:junit. The argument generator is found through ServiceLoader, overridable per test with `@FuzzTest(arguments = ...)`, and a missing provider fails fast with a message naming jqf-generator-quickcheck. Junit5TrialExecutor replays the lifecycle reflectively so the multi-million-trial loop never re-enters the Jupiter engine; an unmet assumption maps to TestAbortedException. An integration test runs SimpleClassTest through the JUnit 5 executor under real instrumentation and reproduces the JUnit 4 coverage (7 branches), confirming the two run paths explore the same code. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@fuzz

The jqf:fuzz and jqf:repro goals supported only @RunWith(JQF.class) + @fuzz (JUnit 4). Detect the test style from its annotations and dispatch: a @fuzztest method runs through JUnit5FuzzRunner over the plugin-built ZestGuidance, while the JUnit 4 path keeps using GuidedFuzzing unchanged. FuzzTestDispatcher chooses the framework explicitly rather than through ServiceLoader<FuzzFramework> -- the plugin classpath carries both the JUnit 4 and JUnit 5 providers, so a service lookup would be ambiguous -- and normalises the JUnit 4 Result and the engine's FuzzResult so the plugin's pass/fail handling is the same on both paths. A failsafe IT fuzzes a planted-bug @fuzztest, replays the saved input, and runs a JUnit 4 campaign unchanged against the instrumented examples. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@fuzztest

Add jqf-generator-instancio, a second reference provider beside junit-quickcheck, to show ArgumentsGeneratorFactory is generation-library agnostic: the engine and both run paths drive it without depending on Instancio. It builds each parameter with Instancio.of(type).withSeed(seed), where the seed is drawn from the engine's guided byte stream, and depends only on jqf-core and instancio-core -- no JUnit 4, no junit-quickcheck. Instancio 5.x is pinned because it is the last line supporting Java 8+. A @fuzztest end-to-end test resolves the provider through ServiceLoader, so a classpath swap changes generation with no change to the test. The README documents the coarser, seed-based guidance trade-off versus a byte-structured generator. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@fuzztest

Add jqf-generator-jetcheck, a third reference provider after junit-quickcheck and Instancio. Unlike Instancio's single seed, jetCheck reads randomness as a stream of ints, so this provider routes each draw to the engine's guided byte stream: the fuzzer's byte mutations map onto local, structural changes in the value, a better fit for coverage-guided search. jetCheck 0.3.0 exposes GenerationEnvironment.generative(IntSource, int) and a public IntSource, so the provider drives a Generator from JQF's StreamBackedRandom through public API only. It maps the eight primitives (and wrappers), String, UUID, and the java.time and java.sql date-time types to jetCheck generators; other types fail fast. A generator that cannot satisfy its constraint for an input (or yields null) is reported as SkipTrialException. Depends on jqf-core and jetCheck only, with no JUnit 4 and no junit-quickcheck. A @fuzztest end-to-end test resolves the provider through ServiceLoader. The generation helpers are public so a test with its own parameter types can supply a custom ArgumentsGeneratorFactory that reuses them instead of duplicating the generation and the skip guard: JetCheckArgumentsGenerator.builder(...) drives the per-trial loop with an explicit sizeHint, JetCheckArgumentsGeneratorFactory.generatorFor exposes the built-in type map, and JetCheckGeneration.generate produces one guarded value. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@fuzztest

The refactor decoupled the engine from JUnit 4 and junit-quickcheck, but the README still described only the @RunWith(JQF.class) + junit-quickcheck path. State that fuzz tests run under JUnit 4 or JUnit 5, that generators are pluggable through the ArgumentsGeneratorFactory SPI (junit-quickcheck by default, Instancio and jetCheck as alternatives), and that input validity uses JUnit 4's Assume or JUnit 5's Assumptions. Add a JUnit 5 @fuzztest example next to the existing one. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

This was referenced Jul 1, 2026

Public API to run a Generator against a caller-supplied source of randomness JetBrains/jetCheck#7

Closed

Allow supplying a custom org.instancio.Random for stream-driven generation (coverage-guided fuzzing) instancio/instancio#1770

Open

vlsi force-pushed the claude/sleepy-visvesvaraya-b2ac72 branch 5 times, most recently from af9b74d to a9773b6 Compare July 4, 2026 21:26

vlsi force-pushed the claude/sleepy-visvesvaraya-b2ac72 branch from a9773b6 to 3aff8f7 Compare July 5, 2026 10:48

vlsi and others added 8 commits July 9, 2026 10:04

vlsi force-pushed the claude/sleepy-visvesvaraya-b2ac72 branch from 3aff8f7 to b9c7ce9 Compare July 9, 2026 07:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Decouple the JQF/Zest engine from JUnit 4 and junit-quickcheck (#80)#278

Decouple the JQF/Zest engine from JUnit 4 and junit-quickcheck (#80)#278
vlsi wants to merge 8 commits into
rohanpadhye:masterfrom
vlsi:claude/sleepy-visvesvaraya-b2ac72

vlsi commented Jul 1, 2026 •

edited

Loading

Uh oh!

vlsi commented Jul 1, 2026

Uh oh!

rohanpadhye commented Jul 1, 2026

Uh oh!

vlsi commented Jul 1, 2026

Uh oh!

vlsi commented Jul 4, 2026

Uh oh!

rohanpadhye commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

vlsi commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Module layout

Commits

Backwards compatibility

Upstream follow-ups for the generator providers

How to verify

Uh oh!

vlsi commented Jul 1, 2026

Uh oh!

rohanpadhye commented Jul 1, 2026

Uh oh!

vlsi commented Jul 1, 2026

Uh oh!

vlsi commented Jul 4, 2026

Uh oh!

rohanpadhye commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vlsi commented Jul 1, 2026 •

edited

Loading