Skip to content

[DNM] sqlcache: serialize write transactions to prevent SQLITE_BUSY under concurrent writers#1232

Open
rohitsakala wants to merge 1 commit into
rancher:mainfrom
rohitsakala:sqlcache-serialize-writers
Open

[DNM] sqlcache: serialize write transactions to prevent SQLITE_BUSY under concurrent writers#1232
rohitsakala wants to merge 1 commit into
rancher:mainfrom
rohitsakala:sqlcache-serialize-writers

Conversation

@rohitsakala

@rohitsakala rohitsakala commented Jul 7, 2026

Copy link
Copy Markdown
Member

Summary

While debugging rancher/rancher#55229 I found that steve's sqlcache serializes nothing on the write path: every GVK informer runs its own write-goroutine, and they all contend for SQLite's single write lock on the one shared informer_object_cache.db. Under concurrent writers this turns into write-lock starvation and surfaces as:

Error in Store.Update for type X: transaction: begin tx: database is locked (5) (SQLITE_BUSY)

This PR serializes write transactions in-process with a sync.Mutex.

Repro script (gist): https://gist.github.com/rohitsakala/31ae257a9b44f2d30da240e368ef2205

  • Scope: may or may not fully resolve #55229 (root cause not confirmed), but it removes a real write-contention path and is a better lever than retry/busy_timeout. I am currently working on finding the root cause, but regardless of it, this PR is a good improvement.

  • Error code 5: with busy_timeout=120000 + _txlock=immediate, each write does BEGIN IMMEDIATE and, if it can't get the write lock, polls the full 120s×3 (~360s) before failing with code 5 (SQLITE_BUSY).

Reproduction scenarios

Reproduced with the gist script.

GVKs churned (write-goroutines) Concurrent PATCH workers Batch Without mutex With mutex (this PR)
3 90 600 SQLITE_BUSY logged 0 errors
11 140 600 SQLITE_BUSY logged 0 errors
14 40 600 SQLITE_BUSY logged 0 errors
20 20 600 SQLITE_BUSY logged 0 errors

Why the mutex has zero errors

I think this works because writeMu lets only one writer reach SQLite at a time, so the single write lock is always free, meaning there's no stampede and nothing to time out on, so code 5 can't occur. And since Go's sync.Mutex is FIFO, the waiting just moves from SQLite's busy-handler into a FIFO.

@rohitsakala rohitsakala force-pushed the sqlcache-serialize-writers branch from 3d44203 to 36f13bc Compare July 7, 2026 06:06
@rohitsakala rohitsakala marked this pull request as ready for review July 10, 2026 07:25
@rohitsakala rohitsakala requested a review from a team as a code owner July 10, 2026 07:25
@rohitsakala rohitsakala requested a review from aruiz14 July 10, 2026 07:41
@tomleb tomleb self-requested a review July 14, 2026 12:39
@rohitsakala rohitsakala changed the title sqlcache: serialize write transactions to prevent SQLITE_BUSY under concurrent writers [DNM] sqlcache: serialize write transactions to prevent SQLITE_BUSY under concurrent writers Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants