Skip to content

fix(rust_brain): preserve HLC on snapshot restore and gossip receive - #56

Draft
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-a94f
Draft

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#56
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-a94f

Conversation

@cursor

@cursor cursor Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Bug and impact

restore_from_file() and gossip.receive() did not preserve or apply Hybrid Logical Clock (HLC) timestamps introduced in v0.6.0. After snapshot restore, any causal successor write with a pre-crash HLC was rejected with TimestampRegression, causing silent data loss in disaster-recovery and cross-node gossip scenarios.

Concrete trigger:

  1. Node writes remember("k", "v1", hlc=(5000, 10, "nodeA"))
  2. Snapshot is taken and restored on a fresh brain
  3. Restored node gets a fresh HLC (wall, 0, uuid) instead of (5000, 10, "nodeA")
  4. Causal successor remember("k", "v2", hlc=(5000, 11, "nodeA")) raises TimestampRegression

Root cause

restore_from_file() reconstructed MemoryNode without the hlc field, falling back to _hlc.now(). gossip.receive() similarly omitted hlc/ts_ns from incoming events.

Fix

  • Add _parse_hlc() for wire/snapshot normalisation (with legacy ts_ns fallback)
  • Restore hlc field and advance global HLC during restore_from_file()
  • Pass hlc through bulk_write and gossip.receive
  • Skip gossip updates missing HLC on existing keys (cannot establish causal order)
  • Hold lock during restore to prevent concurrent writes

Validation

  • Reproduced bug before fix (successor write fails after restore)
  • Added regression tests: test_restore_preserves_hlc_for_replication, gossip HLC ordering tests
  • Full suite: 191 passed, 8 skipped
Open in Web View Automation 

restore_from_file() dropped hlc from snapshots and assigned fresh
_hlc.now() timestamps, causing TimestampRegression on causal successor
writes and silent gossip data loss after disaster recovery.

- Add _parse_hlc() for wire/snapshot normalisation
- Restore hlc field and advance global HLC during restore
- Pass hlc through bulk_write and gossip.receive
- Add regression tests for restore round-trip and gossip ordering

Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant