Concurrent first replica ID creation can fork logstream provenance #2274

Summary

get_replica_id() is not safe when two hub/CLI processes initialize the same fresh palace concurrently. Contenders share replica.json.tmp and each unconditionally replaces replica.json, so callers can fail or return an ID different from the durable identity.

This breaks the function's core invariant: one stable replica ID must stamp every op authored by that palace. A returned-but-not-persisted ID can fork logstream provenance.

Reproduction

On current develop (06cb6987f02610784fefbad4b2bd5d026d164ba6):

1. Start multiple fresh processes against one empty palace.
2. Patch _mint() in each process to wait on a shared barrier after the missing-file check and return a distinct valid ID.
3. Call get_replica_id() in every process.

A four-process run reliably produces FileNotFoundError contenders at the shared replica.json.tmp. A deterministic two-contender run also demonstrated a successful caller returning one ID while replica.json held the other contender's ID.

The race is at:

if path.exists():
    ...
tmp = path.with_suffix(".json.tmp")
tmp.write_text(...)
os.replace(tmp, path)
return replica_id

Expected behavior

Exactly one first creator should publish the canonical identity without overwriting another creator. Every successful contender should return the ID stored in replica.json; the file must never be observable half-written.

Proposed scope

• Write each candidate to a unique, same-directory temporary file.
• Flush it completely before publication.
• Publish with an atomic no-clobber operation.
• On a lost race, validate and return the winner.
• Preserve existing legacy-ID and corrupt-file fail-loud behavior.
• Add a real multi-process regression plus collision/error cleanup tests.

A hard-link-based no-clobber publication fails safely on filesystems that do not support hard links; a portable fallback would need equivalent serialization and crash safety rather than reverting to overwrite semantics.