fix(store): refactor Import() with LWW conflict resolution#267
Open
Snakeblack wants to merge 1 commit intoGentleman-Programming:mainfrom
Open
fix(store): refactor Import() with LWW conflict resolution#267Snakeblack wants to merge 1 commit intoGentleman-Programming:mainfrom
Snakeblack wants to merge 1 commit intoGentleman-Programming:mainfrom
Conversation
Replace blind INSERT loops in Import() with sync_id-keyed conflict resolution: observations use Last-Write-Wins by updated_at, prompts skip-if-exists (immutable). deleted_at is treated as part of the LWW state so soft-deletes propagate and restorations work. - store: ImportResult expanded to granular counters (Inserted / Updated / Skipped per entity). - store: backfill SQL and inline migration use randomblob(8) so every newly generated sync_id matches newSyncID's 16 hex format. - sync: ImportResult mirrors the new granular fields and aggregates them across chunks. estimateMutationImportResult feeds the new Inserted counters. - cmd: 'engram import' and 'engram sync' print the new counters. - tests: 9 new tests covering idempotency, LWW update/skip, immutable prompts, deleted_at preservation/propagation/ restoration, granular counter contract, and newSyncID format consistency. Closes Gentleman-Programming#244
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary / Resumen
Replaces the blind
INSERT-only loops inStore.Import()withsync_id-keyed conflict resolution. Observations now resolve via Last-Write-Wins byupdated_at; prompts use skip-if-exists (they are immutable).deleted_atis treated as part of the LWW state so soft-deletes propagate across machines and active rows can be restored when an importer's payload is newer.Changes
Store Layer
Import()rewritten:SELECT id, updated_at WHERE sync_id = ?→INSERT/UPDATE(full row incl.deleted_at) /SKIPbased onpayload.UpdatedAt > local.UpdatedAt.SELECT id WHERE sync_id = ?→INSERTif absent,SKIPotherwise.ImportResultexpanded to granular counters:SessionsImported,ObservationsInserted/Updated/Skipped,PromptsInserted/Skipped.873,884,5230) now usesrandomblob(8)so every newly generatedsync_idmatchesnewSyncID's 16-hex-char format. Existing IDs in either format are preserved bynormalizeExistingSyncID.Sync Layer
internal/sync.ImportResultmirrors the new granular fields and aggregates them across chunks.estimateMutationImportResultfeeds the newInsertedcounters (best-effort estimate from mutation entities; LWW happens inside the store).CLI
engram importandengram syncprint the new counters.Tests (9 new)
TestImportLWWIdempotentReimport— re-importing the same dump produces zero inserts and N skips.TestImportLWWUpdatesNewerObservation— payload with newerupdated_atupdates the local row.TestImportLWWSkipsOlderObservation— payload older than local is skipped, local content preserved.TestImportLWWPromptsAreImmutable— re-importing a prompt with edited content does not overwrite local.TestImportLWWPreservesDeletedAtOnInsert— soft-deleted obs imported into empty store keepsdeleted_at.TestImportLWWSoftDeletePropagatesViaLWW— payload soft-delete with newerupdated_atpropagates locally.TestImportLWWRestorationViaLWW— payload reactivation with newerupdated_atclearsdeleted_at.TestImportResultHasGranularCounters— asserts the new struct shape.TestNewSyncIDFormatConsistency—newSyncID(prefix)always returnsprefix-{16 hex chars}.Existing tests using the old
ObservationsImported/PromptsImportedfields were updated to read the new*Insertedcounters.Migration impact
UNIQUE(sync_id)constraint added.sync_idvalues (16-hex from Go, 32-hex from old backfill) are accepted as-is;normalizeExistingSyncIDkeeps coexistence working.sync_id— already-backfilled rows are untouched.ImportResult.ObservationsImported/PromptsImportedmust update to the new field names. Updated in this PR forcmd/engramandinternal/sync.Closes #244