Skip to content

Persist the SNARK test caches across CI runs #3467

Description

@jpraynaud

Why

Neither the keygen cache nor the SRS is persisted across CI runs (the toolchain-and-cache action only saves dependency artifacts through Swatinem/rust-cache), so every nightly and every slow PR leg starts cold.

What

Add an actions/cache layer for target/mithril-circuit-test-cache with a three-layer invalidation design: content-keyed directories for semantic staleness, a versions-scoped cache key for serialization format drift, and self-healing readers as backstop.

How

Paths are relative to mithril-stm/src/ unless they start with .github/.

  • Add an actions/cache/restore step for target/mithril-circuit-test-cache in the test job of .github/workflows/test-rust.yml (next to the toolchain-and-cache composite step), with an exact key folding a manual generation token and a hash of the midnight-* entries of Cargo.lock
  • Scope every restore-keys prefix to that versions hash so a midnight-proofs upgrade (serialization format drift) can never restore stale bytes
  • Save with actions/cache/save only from the runs that execute the full slow suite (the include-slow-tests: true invocation from nightly-dispatcher.yml); PR runs stay restore-only
  • Make the readers self-healing: on deserialization failure, KeyProvider::cache_state and read_optional (circuits/key_provider.rs) and TrustedSetupProvider::get_trusted_setup_parameters (circuits/trusted_setup.rs) should delete the offending file and regenerate instead of failing the test
  • Prune fingerprint directories not touched by the current run before saving, so retired fingerprints do not accumulate in the archive
  • Make one recursive verifying-key cache canonical for both the circuit tests and IvcSnarkProverSetup::build_for_test_degree, with the prover path storing only the additional proving key: today proof_system/ivc_halo2_snark/prover_setup.rs and circuits/halo2_ivc/tests/common/generators/setup.rs hold the same key under two protocols, because KeyProvider::verification_key treats a missing proving key as a miss and regenerates the pair
  • Assess the gain (expected: removes the cold start from warm CI runs) and report it in the issue

Depends on #3433.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions