core: Implement initial support for multi-process database access#6236
Open
PThorpe92 wants to merge 20 commits intotursodatabase:mainfrom
Open
core: Implement initial support for multi-process database access#6236PThorpe92 wants to merge 20 commits intotursodatabase:mainfrom
PThorpe92 wants to merge 20 commits intotursodatabase:mainfrom
Conversation
9b7c27c to
7d6cd8c
Compare
Merging this PR will not alter performance
Comparing Footnotes
|
608bbb4 to
94ef7f5
Compare
dd94399 to
6a1b724
Compare
557828e to
d1a18a4
Compare
Move platform specific byte range file locking to IO trait and out of storage
…al schema/header state
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a process-shared WAL coordination layer for Unix 64-bit builds.
Instead of keeping all WAL coordination state inside one process (WalFileShared), we now create a separate
-tshmfile, memory-mapped by every process that opens the database. It stores:max_frame, salts, checksums, checkpoint generation, transaction count)How it works:
-tshmfile.-tshm.-tshmso checkpoints in other processes know how far they are allowed to backfill.-tshmand appends page-to-frame mappings into the shared frame index.-tshmfile.How this differs from SQLite's
-shmfile:On Linux, Turso uses OFD byte-range locks. On macOS and other Unix targets, it falls back to regular process-scoped
fcntllocks plus stored owner PIDs and some extra per-process bookkeeping.Turso's reopen path is more conservative than SQLite's current
-shmrecovery model. It validates that the saved snapshot still matches the WAL file and may fall back to a full WAL scan instead of blindly trusting the persisted shared state. More work is still needed in the entire commit/checkpoint/recovery path to definitively trustnbackfills > 0in an existing-tshmfile and prevent scanning the WAL file to build the index.