Job fetch index tuning and vacuum monitoring - #885
Merged
Conversation
job_i5 was ordered by the fetch's filter, (name, start_after), while the fetch
orders by priority desc, created_on, id. Any queue with a real backlog therefore
read every eligible row and sorted to return the top N — 23-33 ms per fetch at
50k due, on the hottest path in the library.
CREATE INDEX job_i5 ON job (name, priority DESC, created_on, start_after)
WHERE state < 'active' AND NOT blocked
Two details are load-bearing. start_after is a trailing KEY column, not INCLUDE
and not absent: a non-leading key column is still an Index Cond, so not-yet-due
rows are filtered inside the index, and INCLUDE cannot help because FOR UPDATE
forces an Index Scan. And id is deliberately not a key column — it is a random
uuid, so it would defeat btree deduplication and cost 8.5x the index size; the
planner covers the ordering with an Incremental Sort instead (PG13+, pg-boss's
minimum, verified on 13.23).
Measured end-to-end with 8 workers contending through SKIP LOCKED, draining
20,000 jobs: 2,506/2,430 ms -> 701/705 ms, roughly 8,000 -> 28,500 jobs/s.
Insert throughput unaffected, index +4.7% on realistic interleaved multi-queue
data. CockroachDB ~7x faster, YugabyteDB neutral, so no backend gating is
needed. key_strict_fifo is neutral — that fetch never reaches this index.
One shape everywhere: no queue option and no per-queue variance, so drift
detection needs no new concepts. The migration reshapes job_i5 across job_common
and every partition through the existing BAM machinery as a CONCURRENTLY
drop-then-rebuild pair, reversibly, the way v33 did.
For the release notes: the upgrade enqueues 2(N+1) background commands. The DDL
is cheap (832 ms for 52 commands over 25 partitions holding 500k jobs) but the
wall clock is bound by bamIntervalSeconds — about 52 minutes at the 60s default.
Background and non-blocking; the old index stays until its own drop runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFPiCZGB6dmV6mgHWKT2mF
Both were requested as performance escapes from the fetch's sort, not as ordering semantics: orderByCreatedOn's own JSDoc said "improve performance when the order of jobs does not matter", and the commit that fixed the other's SQL emission calls it "fix priority bypass" (a7334db, 10.0.3). The fetch index is now ordered to match the fetch, so there is nothing left to escape. Measured on a 50k-due backlog, they are now pointless to harmful: default 23.9 ms -> 0.092 ms priority: false 16.1 ms -> 16.35 ms (only shape still full-sorting) orderByCreatedOn: false 24.2 ms -> 0.077 ms (saves ~0.015 ms = noise) priority: false is ~180x slower than the default it was added to beat, because no index leads with created_on. Both are now accepted and ignored, with a deprecated_fetch_option warning emitted once per option per instance — only when explicitly set false, since work() forwards both keys with their defaults on every poll. They are marked @deprecated and will be rejected in the next major. The ORDER BY becomes unconditional, which also drops the trailing j.id. id is a random uuid, so it never provided creation order: a batch insert shares one now() and therefore ties on created_on, and those ties resolve today in random uuid order. Without it the index satisfies the ordering outright rather than through an Incremental Sort — 0.097 -> 0.031 ms and 50 -> 5 buffers at limit=1 — and ties fall back to index order, which tracks insertion order better than a uuid does. key_strict_fifo keeps its own id tiebreak in strict_fifo_heads, where DISTINCT ON needs a total order to pick a deterministic head per key. Behaviour change for the release notes, not a deprecation line: priority: false changes dequeue order for anyone who set it AND assigns job priorities. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AFPiCZGB6dmV6mgHWKT2mF
…ation
Every migration was accreting its own test — the v33 job_i5/job_i9 reshape, the
v38 key_strict_fifo head index and its two rollback cases, the v33-pinned BAM
clearing test. That grows without bound and means writing a test for every
schema bump.
Everything is now keyed off MIGRATION_DEPTH (default 3, env-tunable, clamped to
the migration floor), counted back from currentSchemaVersion, so a new migration
is covered the moment it lands and nothing needs editing:
- converges on the fresh-install schema after rolling back N and replaying,
generated per depth 1..N. Creates a partitioned queue per policy first, so
per-partition fan-out and policy-scoped index builds are covered rather than
just job_common — without those a migration that built the wrong index on
the wrong partition would round-trip clean.
- enqueues async work for the newest async migration and clears it on
rollback, discovering that migration from getAll() rather than naming it.
- drops every index a migration created when that migration is rolled back.
That last one exists because the round-trip has a structural blind spot: an
uninstall that fails to drop what its install created is invisible to a
before/after comparison, since replaying recreates it idempotently and the end
states match. Verified by sabotaging v38's uninstall — the round-trip passed and
one of the tests being deleted here caught it. So it reads index names out of
each migration's own DDL via bamCommandIndexName, rolls back, and asserts none
survive. Two traps it has to handle: migrations name job_iN while the catalog
holds job_common_iN, so it matches on the suffix; and CREATE FUNCTION bodies are
excluded, since a migration that only re-declares createQueueFn carries a CREATE
INDEX for every index in the schema and would report the whole set as leaked.
driftTest's job_i5 assertions become shape-based rather than literal, and the
missing-index repair DDL is now verified by executing it and re-checking drift.
job_i5 is the one index that keeps getting reshaped, and spelling its column list
into three tests means every index change edits them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFPiCZGB6dmV6mgHWKT2mF
# Conflicts: # docs/api/events.md # src/types.ts
# Conflicts: # test/driftTest.ts # test/migrationTest.ts
The generic check asserted that every index a migration creates is gone once that migration is rolled back. v40 breaks the premise: it reshapes job_i5 rather than introducing it, so its uninstall drops and rebuilds the index in the previous shape and job_i5 is *supposed* to survive. An index the uninstall also creates is being reshaped, not introduced, so it is now excluded. Re-verified against the original sabotage (v38's uninstall with its DROP INDEX job_i10 removed): still caught, still names the right index and version, so the exemption has not neutered the check. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AFPiCZGB6dmV6mgHWKT2mF
|
coverage: 100.0%. remained the same — indexing-monitoring-26 into master |
kibertoad
added a commit
to kibertoad/pg-boss
that referenced
this pull request
Sep 3, 2026
Upstream released 12.30.0 with migration v40 (job fetch index tuning and vacuum monitoring, timgit#885), the same version slot this branch had taken for schedule.last_job_id. Upstream's v40 stands as written. schedule.last_job_id moves to v41, release 12.31.0, with package.json (version, pgboss.schema) and src/schema.json following it. The timekeeper comment about send-it payloads written without a `key` now names 12.31.0 as the release that added the field. The two commits below still read "Migration v40" and "version 12.30.0" in their messages, since they are already published; the migration is v41.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Righting a wrong on the fetch index that's been around for forever. Then, doing what we can to keep things operating smoothly at scale, including detecting when a noisy server may interfere with vacuum. Partly inspired by https://planetscale.com/blog/keeping-a-postgres-queue-healthy.