Use PostgreSQL template database for faster test resets#1040
Use PostgreSQL template database for faster test resets#1040rowanseymour wants to merge 3 commits intomainfrom
Conversation
resetDB now drops mailroom_test and recreates it from a mailroom_test_tpl template database (built once on startup) instead of running pg_restore on every reset. PostgreSQL clones a template at the file-system level, which is dramatically faster than parsing and reinserting the dump. Locally, the full ./... suite goes from ~2m53s to ~1m53s (~35% faster); core/models alone goes from 25.6s to 5.6s. After regenerating the dump, drop both mailroom_test and mailroom_test_tpl so the next test run rebuilds them.
There was a problem hiding this comment.
Pull request overview
This PR speeds up Go test database resets by switching testsuite.resetDB from pg_restore-based resets to cloning a PostgreSQL template database, reducing suite runtime by avoiding repeated restore/reindex work.
Changes:
- Build a
mailroom_test_tpltemplate DB on first test runtime startup (when missing). - Update
resetDBto dropmailroom_testand recreate it viaCREATE DATABASE ... TEMPLATE mailroom_test_tpl. - Add helper functions for admin DB access and for reopening the sqlx pool after cloning.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| var exists bool | ||
| err := admin.QueryRow(`SELECT EXISTS(SELECT 1 FROM pg_database WHERE datname = $1)`, templateDBName).Scan(&exists) | ||
| require.NoError(t, err) | ||
| return exists |
There was a problem hiding this comment.
templateDBExists only checks for the presence of a database named mailroom_test_tpl, but doesn’t verify it’s correctly configured as a safe clone source (e.g. datistemplate and/or disallowing connections). If the DB exists but is misconfigured or was modified, resets will silently clone bad state. Consider querying pg_database for datistemplate/datallowconn (or equivalent) and recreating the template when it’s not in the expected state.
CI saw `CREATE DATABASE ... TEMPLATE` fail with "source database is being accessed by other users" on the first run when no template existed yet. Closing rt.DB isn't always sufficient — earlier setup paths can leak pools and lib/pq may leave idle connections behind. Terminate any other backends to mailroom_test before cloning.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1040 +/- ##
==========================================
- Coverage 52.45% 52.26% -0.19%
==========================================
Files 275 275
Lines 12682 12727 +45
==========================================
Hits 6652 6652
- Misses 5188 5233 +45
Partials 842 842 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
PR feedback: datistemplate=TRUE alone doesn't block connections, it only gates non-superuser template usage. Also set datallowconn=false so the template really can't be accidentally connected to and modified.
Summary
resetDB(used byReset(t, rt, ResetAll|ResetDB)) now dropsmailroom_testand recreates it from amailroom_test_tpltemplate database instead of runningpg_restoreon every reset. PostgreSQL clones a template at the file system level, which is dramatically faster than parsing the dump and reinserting/reindexing.The template is built once on first run (or when missing), in
Runtime()startup, by cloning the freshly-loadedmailroom_test. After regenerating the dump, drop bothmailroom_testandmailroom_test_tplso the next test run rebuilds them.Local timings (devcontainer,
go test -count=1 -p=1 ./...):core/modelscore/runner/handlersNo test files were touched. The same set of pre-existing flaky tests (
TestMsgReceivedTask,TestOptinRequestedin isolation,TestBroadcastsFromEventsin some orderings) fail on both branches identically.Test plan
go test -count=1 -p=1 ./core/models/passesgo test -count=1 -p=1 ./core/runner/handlers/passesgo test -count=1 -p=1 ./web/...passesgo test -count=1 -p=1 ./...produces the same set of failures as main