Skip to content

fix(rust_brain): preserve HLC on snapshot restore and gossip receive - #57

Draft
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-b2f2
Draft

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#57
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-b2f2

Conversation

@cursor

@cursor cursor Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Bug and impact

restore_from_file() dropped hlc from snapshots and assigned fresh _hlc.now() timestamps. After disaster recovery, any causal successor write (or gossip replay with pre-crash HLC) raises TimestampRegressionsilent data loss in replication/gossip paths.

Trigger scenario: Node crashes with memory at hlc=(5000, 10, "nodeA"). After snapshot restore, a peer gossips hlc=(5000, 11, "nodeA") or the node itself writes a causal update. The write is rejected because restore assigned a fresh wall-clock HLC.

Root cause

v0.6.0 added HLC monotonic enforcement but restore_from_file(), bulk_write(), and gossip.receive() never preserved or forwarded HLC timestamps.

Fix

  • Add _parse_hlc() for wire/snapshot normalisation (with legacy fallback for pre-v0.6.0 snapshots)
  • Restore hlc field and advance global HLC during restore (under lock)
  • Pass hlc through bulk_write and gossip.receive
  • Gossip: skip updates missing HLC when key already exists; reject stale HLC overwrites

Validation

  • Reproduced bug: restore assigned fresh HLC; causal write raised TimestampRegression
  • New regression tests: test_restore_preserves_hlc_for_replication, gossip HLC ordering tests
  • Full suite: 191 passed, 8 skipped
Open in Web View Automation 

restore_from_file() dropped hlc from snapshots and assigned fresh
_hlc.now() timestamps, causing TimestampRegression on causal successor
writes and silent gossip data loss after disaster recovery.

- Add _parse_hlc() for wire/snapshot normalisation
- Restore hlc field and advance global HLC during restore
- Pass hlc through bulk_write and gossip.receive
- Add regression tests for restore round-trip and gossip ordering

Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant