Skip to content

Flaky CI on hydra run of ci/hydra-build:x86_64-linux.mithril-common #3329

Description

@perturbing

Why

All tests that call setup_temp_directory_for_signer share the same base path:

mithril_test/tests_setup/mithril_crypto_helper_material/{party_id}

KES key files in this shared directory are written non-atomically: to_file first calls fs::File::create, which truncates the file to empty, and then writes the JSON content in a second step.

A concurrent test that reads the file between those two steps sees an empty file and panics with:

EOF while parsing a value at line 1 column 0

The result is that some CI runs fail this test due to race conditions (see, for example, this one on an unrelated part of the code base)

What

The test certificate_chain::certificate_verifier::tests::verify_genesis_certificate_fails_if_is_not_genesis, and potentially any other test that reads KES key files from the shared temp directory, fails intermittently when the Hydra runner executes tests in parallel.

Rerunning the job succeeds, masking the underlying data race.

How

  • In fixture_builder.rs, replace the direct to_file calls for kes.sk and opcert.cert with a write-to-temp-then-rename pattern.
  • Use std::process::id() or a random suffix in the temp filename to avoid collisions between parallel processes.
  • fs::rename is atomic on POSIX filesystems, so concurrent readers will always see either no file or a complete one.
  • Verify the same pattern is not needed elsewhere by auditing other callers of to_file that write into a shared temp directory.
  • Add a comment at the write site explaining why the atomic rename is necessary.

Metadata

Metadata

Assignees

Labels

bug ⚠️Something isn't working

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions