Allow supplying a custom org.instancio.Random for stream-driven generation (coverage-guided fuzzing)

[vlsi]

Use case

Coverage-guided fuzzers such as JQF/Zest drive a generator from a mutable byte stream: the fuzzer mutates the raw bytes, the generator turns them into a structured object, and coverage feedback steers the next mutation. For this to work well, the generator has to consume the stream incrementally, so that a small byte change produces a small, local change in the object (one field, one list element) rather than a completely different graph.

Current state

Instancio derives all randomness from a single seed via withSeed(long). That is ideal for reproducibility, but coarse for a fuzzer: one seed determines the entire object graph, so flipping one byte of the fuzzer's input reshapes everything at once instead of one field. Coverage-guided search cannot map input regions onto structure.

There is no public way to supply a custom source of randomness:

• InstancioApi exposes only withSeed(long) and withSettings(Settings).
• org.instancio.spi.InstancioServiceProvider has providers for generators, type resolution, instantiation, setter resolution, and annotations — but none for randomness.
• org.instancio.Random is a public interface, yet nothing lets a caller inject an implementation of it; the internal Random is always built from the seed.

I integrated Instancio as a pluggable generator provider in JQF using the seed approach, and documented the trade-off in the module README (linked below): Zest can only steer generation through the eight-byte seed.

What would help

org.instancio.Random already exposes exactly the right shape for a byte-driven implementation — intRange, longRange, stringOf, characterRange, oneOf, trueOrFalse, and so on. Each call naturally consumes a few bytes from a stream. The only missing piece is a way to inject a custom implementation. For example:

// Option A: a builder hook, parallel to withSeed/withSettings
public InstancioApi<T> withRandom(org.instancio.Random random);

// Option B: a provider in the existing SPI
interface InstancioServiceProvider {
    default RandomProvider getRandomProvider() { return null; }
}

With either, a fuzzer supplies a Random backed by its byte stream, and Instancio's generation becomes byte-structured and steerable, while withSeed stays the default for everyone else. This is the property that lets a coverage-guided fuzzer exploit the generator, and it is the one thing keeping Instancio a coarser fit than a byte-structured library for this use case.

Links

• Integration PR: Decouple the JQF/Zest engine from JUnit 4 and junit-quickcheck (#80) rohanpadhye/JQF#278
• Instancio provider README (documents the current seed-based trade-off): https://github.com/vlsi/jqf/blob/163cdc26067460763ed65782f853290e4bdca08e/jqf-generator-instancio/README.md
• The generator that reads a seed from the fuzzer's stream today: https://github.com/vlsi/jqf/blob/163cdc26067460763ed65782f853290e4bdca08e/jqf-generator-instancio/src/main/java/edu/berkeley/cs/jqf/fuzz/instancio/InstancioArgumentsGenerator.java

Thanks for Instancio — the public Random interface is already most of the way there; a way to plug it in would make Instancio a first-class option for coverage-guided fuzzing.

[armandino]

@vlsi thank you for the suggestion and providing the details. If I understood correctly, you need a hook to inject a Random instance for finer-grained control of generated data rather than relying on providing a long seed. I think it should be doable, but I will need to spend some time on this and experiment a little.

Of the two options you mentioned, I'd probably lean toward a withRandom() method rather than adding an InstancioServiceProvider hook, e.g.

withRandom(org.instancio.Random)

or using the RandomGenerator interface added in Java 17, which will be the baseline starting from Instancio v6:

withRandom(java.util.random.RandomGenerator)

org.instancio.Random declares a lot of methods, so RandomGenerator might be a more user-friendly option (and also lower risk in terms of future breaking changes).

I will look into this in more detail once I have some time. In the meantime, please let me know if you have any comments on the above.

[vlsi]

org.instancio.Random

I realized it would be suboptimal from the end-user perspective.
jqf/zest needs full control over the generated random bytes.

java.util.random.RandomGenerator would be reasonable.
jqf/zet would then use java.util.Random subclass that fetches random bytes from a pre-generated byte sequence (see https://github.com/rohanpadhye/JQF/blob/81ce371e28cab8956e08a5a37409451411a8511d/fuzz/src/main/java/edu/berkeley/cs/jqf/fuzz/guidance/StreamBackedRandom.java#L52)

[armandino]

Hey @vlsi, after looking into it a bit more, I'm leaning towards a different approach, and wanted to check if this will work for you.

Instead of adding a new withRandom() API on the builder chain, I was thinking of providing a Hooks class with the following usage:

Hooks.setRandomGenerator(java.util.Random());

// all Instancio.create() calls will use the supplied Random

Hooks.clear();

Hooks would be an undocumented "feature" not intended for public use, but it could evolve into something else down the road. At this point, I'm just not sure about committing to a new top-level API, as it seems to be a niche use case.

If you think this might work for you I can put together a PR so you can try it out. WDYT?

[vlsi]

Frankly, a pseudo-private Hooks.setRandomGenerator does not differ much from a method with explicit @API(status=EXPERIMENTAL, ...) from my point of view.

---

Fuzzers and PBT (property-based testing) might favour different aspects.
For "regular" PBT generator fidelity might be slightly more important.

For instance, for PBT might want using Java's Random.nextInt(int bound) which internally performs rejection sampling to make output uniform.
For PBT, you might want using jumpable random generators to reduce bias in map/flatmap.

On contrary, for fuzzers, it is important to avoid "non-significant" bits. That means, Java's rejection sampling draws multiple ints from the entropy source, and it becomes harder for the fuzzer to steer the generator.

---

My understanding is that:

1. You probably don't want implementing random generator from externally-provided entropy source.
   That means you'd better consume ~java.util.Random which is provided by the users (e.g. fuzzers or PBT users).

2. long seed is too weak for PBT. Problem space for long a, b; a+b is 128 bits, and random-based generator that uses a single 64bit will cover only a fraction of the problem space.
   See Consider stronger random generators like L128X1024MixRandom #1783.