Skip to content

fix(store): refactor Import() with LWW conflict resolution#267

Open
Snakeblack wants to merge 1 commit intoGentleman-Programming:mainfrom
Snakeblack:feat/sync-id-resolution
Open

fix(store): refactor Import() with LWW conflict resolution#267
Snakeblack wants to merge 1 commit intoGentleman-Programming:mainfrom
Snakeblack:feat/sync-id-resolution

Conversation

@Snakeblack
Copy link
Copy Markdown

Summary / Resumen

Replaces the blind INSERT-only loops in Store.Import() with sync_id-keyed conflict resolution. Observations now resolve via Last-Write-Wins by updated_at; prompts use skip-if-exists (they are immutable). deleted_at is treated as part of the LWW state so soft-deletes propagate across machines and active rows can be restored when an importer's payload is newer.

Changes

Store Layer

  • Import() rewritten:
    • Observations: SELECT id, updated_at WHERE sync_id = ?INSERT / UPDATE (full row incl. deleted_at) / SKIP based on payload.UpdatedAt > local.UpdatedAt.
    • Prompts: SELECT id WHERE sync_id = ?INSERT if absent, SKIP otherwise.
  • ImportResult expanded to granular counters: SessionsImported, ObservationsInserted/Updated/Skipped, PromptsInserted/Skipped.
  • Backfill SQL (lines 873, 884, 5230) now uses randomblob(8) so every newly generated sync_id matches newSyncID's 16-hex-char format. Existing IDs in either format are preserved by normalizeExistingSyncID.

Sync Layer

  • internal/sync.ImportResult mirrors the new granular fields and aggregates them across chunks.
  • estimateMutationImportResult feeds the new Inserted counters (best-effort estimate from mutation entities; LWW happens inside the store).

CLI

  • engram import and engram sync print the new counters.

Tests (9 new)

  • TestImportLWWIdempotentReimport — re-importing the same dump produces zero inserts and N skips.
  • TestImportLWWUpdatesNewerObservation — payload with newer updated_at updates the local row.
  • TestImportLWWSkipsOlderObservation — payload older than local is skipped, local content preserved.
  • TestImportLWWPromptsAreImmutable — re-importing a prompt with edited content does not overwrite local.
  • TestImportLWWPreservesDeletedAtOnInsert — soft-deleted obs imported into empty store keeps deleted_at.
  • TestImportLWWSoftDeletePropagatesViaLWW — payload soft-delete with newer updated_at propagates locally.
  • TestImportLWWRestorationViaLWW — payload reactivation with newer updated_at clears deleted_at.
  • TestImportResultHasGranularCounters — asserts the new struct shape.
  • TestNewSyncIDFormatConsistencynewSyncID(prefix) always returns prefix-{16 hex chars}.

Existing tests using the old ObservationsImported / PromptsImported fields were updated to read the new *Inserted counters.

Migration impact

  • No DB schema migration. No UNIQUE(sync_id) constraint added.
  • Existing sync_id values (16-hex from Go, 32-hex from old backfill) are accepted as-is; normalizeExistingSyncID keeps coexistence working.
  • Backfill format change only applies to NEW rows that go through the migration with a NULL/empty sync_id — already-backfilled rows are untouched.
  • Public API change: callers reading ImportResult.ObservationsImported / PromptsImported must update to the new field names. Updated in this PR for cmd/engram and internal/sync.

Closes #244

Replace blind INSERT loops in Import() with sync_id-keyed conflict
resolution: observations use Last-Write-Wins by updated_at,
prompts skip-if-exists (immutable). deleted_at is treated as part
of the LWW state so soft-deletes propagate and restorations work.

- store: ImportResult expanded to granular counters (Inserted /
  Updated / Skipped per entity).
- store: backfill SQL and inline migration use randomblob(8) so
  every newly generated sync_id matches newSyncID's 16 hex format.
- sync: ImportResult mirrors the new granular fields and aggregates
  them across chunks. estimateMutationImportResult feeds the new
  Inserted counters.
- cmd: 'engram import' and 'engram sync' print the new counters.
- tests: 9 new tests covering idempotency, LWW update/skip,
  immutable prompts, deleted_at preservation/propagation/
  restoration, granular counter contract, and newSyncID format
  consistency.

Closes Gentleman-Programming#244
Copilot AI review requested due to automatic review settings April 27, 2026 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(store): refactor Import() with LWW conflict resolution to prevent blind overwrites

1 participant