Fix/wal growth and goroutine leak

[gehrkefc]

No description provided.

[gehrkefc]

Investigation Summary & Test Results

This PR addresses two issues observed in steve v0.7.43 (Rancher 2.13.5):

1. Unbounded WAL growth (~32GB in production)
2. Goroutine leak in watch forwarder loops when downstream disconnects

---

Root Cause Analysis

WAL Growth (Reader Starvation)

SQLite's default auto-checkpoint is PASSIVE — it never waits for readers, and critically, it cannot reset the WAL write pointer when any reader holds a snapshot. In steve, concurrent /v1 list requests create overlapping read transactions. With sufficient concurrency (observed at ~50+ concurrent listers), there is always at least one active reader, so:

• Auto-checkpoint runs at every COMMIT (≥1000 pages)
• It successfully copies frames to the DB file (ckpt == log)
• But it cannot reset/wrap the WAL → file grows linearly forever

This is documented behavior: sqlite.org/wal.html:

"If a database has many concurrent overlapping readers and there is always at least one active reader, then no checkpoints will be able to complete and hence the WAL file will grow without bound."

Goroutine Leak

WatchByPartitions (proxy_store.go) and Watch (store.go) spawn goroutines that forward events via channel send. When the downstream consumer disconnects (context cancelled), these goroutines block forever on result <- item because there's no ctx.Done() select case.

---

What This PR Does

Commit 1: Background WAL Checkpoint

• Adds a periodic PRAGMA wal_checkpoint(TRUNCATE) every 60 seconds with a 30-second timeout
• TRUNCATE mode waits for readers to finish then resets the WAL to 0 bytes
• If the checkpoint times out (stuck reader), it logs a warning and retries next cycle
• Adds Close() method to stop the checkpoint goroutine on shutdown

Commit 2: Goroutine Leak Fix

• Adds select { case result <- item: case <-ctx.Done(): return } in both watch forwarder goroutines
• Ensures goroutines exit when the consumer disconnects

---

Integration Test Results

We created a stress test suite (branch test/sqlite-stress-profiling) that runs steve with a real K8s cluster and measures WAL growth over time.

Test configuration: 10 watchers + 5 writers + 50 concurrent listers, running for 20 minutes.

WITHOUT fix:

Metric	Value
Final WAL	2,076 MB
Peak WAL	2,076 MB
Growth rate	~103 MB/min
Projected 8h	~48 GB
SQLITE_BUSY errors	0
Goroutines	1,355 (stable)
Total writes	80,505
Total lists	544,235

WITH fix (same workload):

Metric	Value
Final WAL	0 MB
Peak WAL	293 MB (briefly, during one checkpoint timeout)
Average WAL	~65 MB
Checkpoint timeouts	1 (logged, recovered next cycle)
Total writes	~80K
Total lists	~541K

The fix eliminates unbounded WAL growth while maintaining the same throughput.

Connection-close test (goroutine leak):

Tested 30 clients abandoning mid-stream + 10 watch disconnects. Goroutines returned to baseline (35) after all contexts cancelled — no goroutine leak with the fix applied.

---

Code Locations

• pkg/sqlcache/db/client.go — WAL checkpoint goroutine, constants, Close() method
• pkg/stores/sqlproxy/proxy_store.go — WatchByPartitions goroutine leak fix
• pkg/stores/sqlpartition/store.go — Watch goroutine leak fix

Related Branches (on gehrkefc/steve fork)

• fix/wal-growth-and-goroutine-leak — this PR
• feature/query-execution-timeout — slow query monitoring (separate concern)
• test/sqlite-stress-profiling — integration tests with profiling