feat(fuzz): introduce continuous fuzzing for oscal catalog (#2080)

[nXtCyberNet]

Summary

This PR introduces a robust continuous fuzzing suite for compliance-trestle using ClusterFuzzLite and Atheris. The primary goal is to improve the security, correctness, and long-term reliability of OSCAL parsing by continuously fuzzing the complex and recursive Catalog data model.

This implementation directly addresses the OpenSSF Scorecard Fuzzing check (currently 0/10), with the intent of raising it to a full 10/10 by enabling automated, coverage-guided fuzzing on every pull request.

Fix #2018

---

Types of changes

• Hot fix (emergency fix and release)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Documentation (change which affects the documentation site)
• Breaking change (fix or feature that would cause existing functionality to change)
• Release (develop -> main)

---

Quality assurance (all should be covered)

• My code follows the code style of this project.
• Documentation for my change is up to date.
• My PR meets testing requirements.
• All new and existing tests passed.
• All commits are signed-off.

---

Key Changes

1. Fuzzing Harness

• Added tests/fuzz/fuzz_catalog.py, a coverage-guided fuzzing harness targeting the OSCAL Catalog model.
• Implements metamorphic (round-trip) testing to ensure that load → serialize → load remains idempotent and lossless, detecting silent data corruption and logic regressions.

2. Structure-Aware Mutation

• Implemented a custom recursive mutator that preserves valid JSON structure while intelligently mutating leaf-node values (strings, UUIDs, integers, nested arrays).
• This allows the fuzzer to bypass superficial validation and exercise deep business logic paths beyond standard schema enforcement.

3. Infrastructure-as-Code for Fuzzing

• Defined the fuzzing build environment using .clusterfuzzlite/Dockerfile.
• Added a build script to package the Python fuzzer for the libFuzzer engine.
• Integrated a GitHub Actions workflow to execute fuzzing for 10 minutes on every PR, enabling continuous fuzzing.

4. Bug Fixes

• Fixed an AttributeError by correctly using Pydantic v1 APIs (parse_raw) for the Catalog model, ensuring the fuzzer runs correctly in the current environment.

---

Testing Instructions

To run the fuzzer locally:

pip install atheris pydantic
python3 tests/fuzz/fuzz_catalog.py

Expected behavior:

• The fuzzer establishes a baseline and begins exploring code paths.
• Coverage growth can be observed in the logs (e.g., NEW_UNIT_ADDED).

---

Key links

• [Sonar coverage](https://sonarcloud.io/dashboard?id=compliance-trestle)

---

Before you merge

• Ensure this PR is merged as a squash commit (if not part of a release).
• Ensure CI is currently passing.
• Review Sonar results. If this is a fork, a maintainer will reach out if additional steps are required.

[degenaro]

@nXtCyberNet Thx for this PR. Would you be able to present this at an upcoming community call? We have one tomorrow at 11AM ET. Short notice, but there is no rush if that is too soon for you, we can have a presentation two weeks hence. If the time zone is an issue, you could present at the APAC community meeting instead. Or both!

See https://oscal-compass.dev/community/#community-meetings-and-communications. Let us know and we can add this topic to the agenda. Thx!

[nXtCyberNet]

Sure, I’ll try to present tomorrow at 11AM ET.

[degenaro]

Sure, I’ll try to present tomorrow at 11AM ET.

Great! I added you to the agenda: #2080 (comment)

I see there a a couple of pipeline failures...

[degenaro]

Fixes (in part) #2018.

[nXtCyberNet]

Sure, I’ll try to present tomorrow at 11AM ET.

Great! I added you to the agenda: #2080 (comment)

I see there a a couple of pipeline failures...

Sure, I’ll try to present tomorrow at 11AM ET.

Great! I added you to the agenda: #2080 (comment)

I see there a a couple of pipeline failures...

sorry , the lint pipeline failed because of the PEP8 style , i will fix it in next commit , and the 2nd was due to the difference in file location, i will correct them asap

[nXtCyberNet]

Hi @degenaro , following up from the community call — happy to elaborate on what PR #2080 is actually adding to the project.

The core idea is round-trip idempotency testing: the fuzzer loads a mutated JSON input into the Catalog model, serializes it back out, and re-parses it. If the two parsed objects don't match, that's a silent data corruption bug — the kind that unit tests almost never catch because they only test inputs we already expect.

What makes this more than just "run a fuzzer and hope for the best" is the structure-aware mutator. Instead of throwing random bytes at the parser (which Pydantic rejects immediately), it preserves valid JSON structure and mutates leaf values — UUIDs, strings, nested arrays, integer fields. This lets it bypass surface-level schema validation and actually exercise the deeper business logic in the Catalog model.

On the infrastructure side, ClusterFuzzLite was chosen specifically because it needs zero external setup — everything runs inside GitHub Actions, corpus included. The fuzz run on PRs is capped at 10 minutes and configured as a non-blocking warning, so it won't slow down normal development.

As for concrete value already delivered: the harness caught a real AttributeError caused by mixing Pydantic v1 and v2 API calls in the runtime environment.

The longer-term goal is OpenSSF Scorecard Fuzzing 10/10 (currently 0/10), which closes issue #2018. The Catalog model is the pilot — once merged, the plan is to generalise the harness and expand to Profile and SSP incrementally.

I know CI is currently failing — that's on my list to fix before requesting a formal review. Happy to jump on a call or answer any questions about the design.

[nXtCyberNet]

@degenaro
I’ve updated the PR to address the CI errors. Could you please re-run the CI pipelines to verify whether the changes resolve the issue?

[nXtCyberNet]

@butler54 @degenaro

Sorry for the delayed response over the last 15 days — life got quite busy ,

I have made significant improvements to the fuzzing harness since the initial review:

• Replaced the dummy seed with the real NIST SP800-53 Rev5 minimal catalog JSON (pinned to a specific commit) as the primary seed. This provides realistic, production-scale OSCAL structure for much better mutation depth and coverage.
• Completely rewrote fuzz_docs.md — the old RFC/Pilot version has been removed. It is now a clean, final "Fuzzing Harness for compliance-trestle — Complete Maintainer Guide" that documents the full architecture, two-seed design, structure-aware mutator, normalisation logic, round-trip idempotency test, trade-offs, troubleshooting, and future roadmap.
• Improved the custom_mutator with better fallback logic, structure preservation, and more effective mutation strategies.
• Enhanced the round-trip idempotency checks for more reliable bug detection (including proper handling of Stage 1 vs Stage 2 exceptions).

I have also addressed the earlier review comments:

• Removed unused code (the old dummy catalogg dict)
• Cleaned up the workflow file (removed unnecessary push trigger, paths, etc.)
• Kept the matrix for future-proofing when we add more sanitizers and models

The harness is now much more robust and focused on finding real silent data corruption bugs through metamorphic testing rather than random invalid inputs.

Could you please take another look at the updated PR when you have time? I'm happy to make any further adjustments or hop on a quick call to discuss.

Thanks!