Skip to content

core: Implement initial support for multi-process database access#6236

Open
PThorpe92 wants to merge 20 commits intotursodatabase:mainfrom
PThorpe92:multiprocess2
Open

core: Implement initial support for multi-process database access#6236
PThorpe92 wants to merge 20 commits intotursodatabase:mainfrom
PThorpe92:multiprocess2

Conversation

@PThorpe92
Copy link
Copy Markdown
Collaborator

This PR adds a process-shared WAL coordination layer for Unix 64-bit builds.

Instead of keeping all WAL coordination state inside one process (WalFileShared), we now create a separate -tshm file, memory-mapped by every process that opens the database. It stores:

  • the current authoritative WAL snapshot (max_frame, salts, checksums, checkpoint generation, transaction count)
  • who currently owns the writer lock, checkpoint lock, and each shared reader slot
  • a shared page-to-frame index so readers can find the newest WAL frame for a page without rescanning the whole WAL file
  • extra metadata used to decide whether a reopened process can trust the saved snapshot

How it works:

  1. On open, Turso decides whether this process is alone or joining an already-open database by probing byte 0 of the -tshm file.
  2. If this is a clean exclusive open, it may repair stale lock ownership left behind by dead processes and reuse or rebuild the shared frame index from the WAL file.
  3. Each writer still uses the existing in-process lock path for other connections in the same process, but it also takes a cross-process writer lock in -tshm.
  4. Each reader still takes the normal local read mark, and also registers a shared reader slot in -tshm so checkpoints in other processes know how far they are allowed to backfill.
  5. When a commit becomes visible, the writer publishes the new WAL snapshot into -tshm and appends page-to-frame mappings into the shared frame index.
  6. Checkpoints consult both the local read marks and the shared reader slots, so they stop before overwriting pages that another process may still need.
  7. Shutdown checkpointing is only allowed when the process can prove it is the last process still mapped to that -tshm file.

How this differs from SQLite's -shm file:

  • On Linux, Turso uses OFD byte-range locks. On macOS and other Unix targets, it falls back to regular process-scoped fcntl locks plus stored owner PIDs and some extra per-process bookkeeping.

  • Turso's reopen path is more conservative than SQLite's current -shm recovery model. It validates that the saved snapshot still matches the WAL file and may fall back to a full WAL scan instead of blindly trusting the persisted shared state. More work is still needed in the entire commit/checkpoint/recovery path to definitively trust nbackfills > 0 in an existing -tshm file and prevent scanning the WAL file to build the index.

Copy link
Copy Markdown

@turso-bot turso-bot bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please review @pedrocarlo

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq bot commented Apr 4, 2026

Merging this PR will not alter performance

✅ 311 untouched benchmarks
⏩ 105 skipped benchmarks1


Comparing PThorpe92:multiprocess2 (c1ddf44) with main (36acc75)

Open in CodSpeed

Footnotes

  1. 105 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@PThorpe92 PThorpe92 force-pushed the multiprocess2 branch 3 times, most recently from 608bbb4 to 94ef7f5 Compare April 5, 2026 21:35
@PThorpe92 PThorpe92 force-pushed the multiprocess2 branch 2 times, most recently from dd94399 to 6a1b724 Compare April 6, 2026 12:41
@PThorpe92 PThorpe92 marked this pull request as ready for review April 6, 2026 14:05
Copy link
Copy Markdown

@turso-bot turso-bot bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please review @pereman2

@PThorpe92 PThorpe92 force-pushed the multiprocess2 branch 5 times, most recently from 557828e to d1a18a4 Compare April 7, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants