Skip to content

Add ordered indexes for job fetches - #1

Draft
bcomnes wants to merge 1 commit into
masterfrom
bret/ordered-job-fetch-indexes
Draft

Add ordered indexes for job fetches#1
bcomnes wants to merge 1 commit into
masterfrom
bret/ordered-job-fetch-indexes

Conversation

@bcomnes

@bcomnes bcomnes commented Jul 22, 2026

Copy link
Copy Markdown
Owner

What

  • add job_i10 (name, priority DESC, created_on, id) for priority-ordered fetches
  • add job_i11 (name, created_on, id) for FIFO fetches
  • keep both indexes partial with the fetch query's state < 'active' AND NOT blocked predicate
  • add schema v38 migration support for existing tables, fresh installs, and future queue partitions
  • keep job_i5 (name, start_after) for queues dominated by future-scheduled jobs
  • include the new indexes in schema drift detection and the index-plan benchmark helper

Why

The existing fetch index can filter by queue and schedule time, but it cannot satisfy either fetch ordering. PostgreSQL may therefore scan and sort the eligible working set before applying LIMIT, and a plan/statistics change can make that work unexpectedly expensive.

The ordered indexes let PostgreSQL walk candidates in the required order and stop as soon as the batch is full. This is independent of any query-shape changes.

Migration impact

  • Standard partitioned PostgreSQL deployments enqueue CREATE INDEX CONCURRENTLY through BAM for job_common and every existing partition.
  • The updated create_queue function gives future partitions the indexes at creation time.
  • Non-partitioned backends create the same indexes directly on their single job table.
  • Completed, active, and blocked rows are excluded, limiting storage and write amplification to fetchable states.
  • Rollback removes both indexes and restores the prior queue-creation function.

Validation

  • npx vitest run: 65 files, 844 tests passed
  • npx eslint .
  • npm run tsc
  • npm run build
  • npm run gen:manifest:check
  • Local PostgreSQL plan check with 200,000 completed and 20,000 eligible rows:
    • priority order selected job_common_i10
    • FIFO order selected job_common_i11
    • neither plan contained a Sort node

@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 29880243419

Coverage decreased (-0.07%) to 97.799%

Details

  • Coverage decreased (-0.07%) from the base build.
  • Patch coverage: 4 of 4 lines across 2 files are fully covered (100%).
  • No coverage regressions found.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 2572
Covered Lines: 2572
Line Coverage: 100.0%
Relevant Branches: 1653
Covered Branches: 1560
Branch Coverage: 94.37%
Branches in Coverage %: Yes
Coverage Strength: 366.11 hits per line

💛 - Coveralls

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants