[DNM] sqlcache: serialize write transactions to prevent SQLITE_BUSY under concurrent writers#1232
Open
rohitsakala wants to merge 1 commit into
Open
[DNM] sqlcache: serialize write transactions to prevent SQLITE_BUSY under concurrent writers#1232rohitsakala wants to merge 1 commit into
rohitsakala wants to merge 1 commit into
Conversation
…ters no longer fail with SQLITE_BUSY
3d44203 to
36f13bc
Compare
aruiz14
approved these changes
Jul 14, 2026
tomleb
approved these changes
Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
While debugging rancher/rancher#55229 I found that steve's sqlcache serializes nothing on the write path: every GVK informer runs its own write-goroutine, and they all contend for SQLite's single write lock on the one shared
informer_object_cache.db. Under concurrent writers this turns into write-lock starvation and surfaces as:This PR serializes write transactions in-process with a
sync.Mutex.Repro script (gist): https://gist.github.com/rohitsakala/31ae257a9b44f2d30da240e368ef2205
Scope: may or may not fully resolve #55229 (root cause not confirmed), but it removes a real write-contention path and is a better lever than retry/
busy_timeout. I am currently working on finding the root cause, but regardless of it, this PR is a good improvement.Error code 5: with
busy_timeout=120000+_txlock=immediate, each write doesBEGIN IMMEDIATEand, if it can't get the write lock, polls the full 120s×3 (~360s) before failing with code 5 (SQLITE_BUSY).Reproduction scenarios
Reproduced with the gist script.
SQLITE_BUSYloggedSQLITE_BUSYloggedSQLITE_BUSYloggedSQLITE_BUSYloggedWhy the mutex has zero errors
I think this works because
writeMulets only one writer reach SQLite at a time, so the single write lock is always free, meaning there's no stampede and nothing to time out on, so code 5 can't occur. And since Go'ssync.Mutexis FIFO, the waiting just moves from SQLite's busy-handler into a FIFO.