Merge upstream/master into feat/schedule-last-job-id

Upstream released 12.30.0 with migration v40 (job fetch index tuning and vacuum
monitoring, timgit#885), the same version slot this branch had taken for
schedule.last_job_id.

Upstream's v40 stands as written. schedule.last_job_id moves to v41, release
12.31.0, with package.json (version, pgboss.schema) and src/schema.json
following it. The timekeeper comment about send-it payloads written without a
`key` now names 12.31.0 as the release that added the field.

The two commits below still read "Migration v40" and "version 12.30.0" in their
messages, since they are already published; the migration is v41.

Author: kibertoad

docs/api/constructor.md

@@ -165,6 +165,46 @@ The following configuration options should not normally need to be changed, but
 
   How often the index bloat check runs. One instance per interval performs it, coordinated through the database, so adding instances does not multiply the work. Cannot exceed 24 hours.
 
+* **monitorVacuum**, bool, default true
+
+  Whether to check that vacuum is keeping up with the queues. Set `false` to disable.
+
+  One measurement, two [`warning`](./events.md#warning) types, because the fixes are opposite:
+
+  | warning | what it means | the fix |
+  | --- | --- | --- |
+  | `xmin_horizon` | vacuum runs and reclaims nothing | find and release whatever is pinning the horizon |
+  | `autovacuum_disabled` | nothing is vacuuming the table at all | turn autovacuum back on, or vacuum on a schedule that keeps up |
+
+  While something holds the horizon back — a backend sitting in an open transaction, a lagging replication slot, a standby with `hot_standby_feedback` enabled, or a prepared transaction — autovacuum cannot reclaim anything your queues delete. Dead tuples accumulate, indexes bloat, and every later vacuum pass gets more expensive. This is the precondition behind most reports of a Postgres queue degrading over time, and it is invisible from the queue's own counters: a backlog caused by too few workers and a backlog caused by a pinned horizon look identical and have opposite fixes.
+
+  There is no threshold to tune, because the check measures the damage rather than guessing at it. It reads `pg_stat_user_tables` for pg-boss's own job tables. Both warnings share a first condition:
+
+  1. a job table is past the point Postgres itself would vacuum it — `autovacuum_vacuum_threshold + autovacuum_vacuum_scale_factor × n_live_tup`, honouring per-table storage parameters over the cluster settings.
+
+  Two consecutive passes then decide which diagnosis applies. For `xmin_horizon`:
+
+  2. a vacuum has since run on that table and the dead-tuple count did not fall, so vacuum tried and reclaimed nothing — this is what separates a pinned horizon from ordinary churn, where a vacuum drops the count sharply;
+  3. a horizon holder exists that is old enough to explain it — for a backend, one whose transaction was already open when that vacuum ran; for a replication slot, standby or prepared transaction, any at all, since those advertise an xmin only while something is genuinely stuck.
+
+  For `autovacuum_disabled`, instead:
+
+  2. the table has `autovacuum_enabled = false`, no vacuum ran between the two passes, and the dead-tuple count grew.
+
+  Turning autovacuum off on a queue table and vacuuming on your own schedule is a legitimate setup, and it stays quiet: a manual vacuum both moves the timestamp and drops the count, so neither branch matches. The warning is for the case where nothing is running at all.
+
+  Sensitivity is therefore tuned with Postgres's own autovacuum settings, per table if you want a particular queue watched more or less closely:
+
+  ```sql
+  ALTER TABLE pgboss.job_common SET (autovacuum_vacuum_scale_factor = 0.05);
+  ```
+
+  Because the second condition compares two observations, the first supervise pass after a horizon is pinned never warns; the warning arrives on a later pass, once a vacuum has actually failed.
+
+  The `xmin_horizon` warning names which holder is responsible so it can be tracked down — start with `pg_stat_activity` for idle-in-transaction backends and `pg_replication_slots` for unread slots. If the connected role cannot read one of those catalogs, the warning's `unreadableSources` lists what could not be checked, so a partial answer is never reported as a clean one.
+
+  Not available on CockroachDB or YugabyteDB, which reclaim on their own schedule rather than from the oldest live snapshot.
+
 * **flowIntervalSeconds**, int, default 5 seconds
 
   How often the background flow resolver runs to unblock dependent jobs (created via [`flow()`](./jobs.md#flowjobs-options)) whose parents have completed. Completing a job no longer unblocks its dependents inline; this resolver handles it shortly after, off the completion hot path. Only runs when `supervise` is enabled.

docs/api/events.md

@@ -33,6 +33,9 @@ boss.on('warning', ({ message, data }) => {
 | `clock_skew` | Database clock is out of sync with application server | `seconds`, `direction` |
 | `listen_notify_unavailable` | `useListenNotify` is enabled but a `LISTEN/NOTIFY` listener could not be established (for example a `db` adapter without `listen`, or PgBouncer transaction pooling); pg-boss continues with polling only | `type`, `error` |
 | `index_bloat` | A job index is holding far more pages than its live entries need and was not rebuilt — because rebuilds are disabled, the connected role does not own the index, the index exceeds `maxIndexBytes`, or `REINDEX CONCURRENTLY` failed. The message names the reason. Emitted once per index rather than on every pass, and again if the condition returns after being cleared | `name`, `table`, `pages`, `entries`, `bytes`, `owned` |
+| `xmin_horizon` | Something is holding the database's MVCC transaction horizon back — an open transaction, a replication slot, a standby with `hot_standby_feedback`, or a prepared transaction — so autovacuum cannot reclaim the rows pg-boss deletes and both tables and indexes grow without bound. Fires on measured evidence: a job table past the point Postgres would vacuum it, a vacuum that has since run and reclaimed nothing, and a holder old enough to explain it (see [`monitorVacuum`](./constructor.md#monitorvacuum)). The message names the holder class and the table it was measured on. Emitted once per episode rather than on every pass, and again if the horizon is pinned a second time | `source`, `holder`, `transactions`, `table`, `tables`, `liveTuples`, `deadTuples`, `budget`, `vacuumAgeSeconds`, `oldestTransactionSeconds`, `unreadableSources` |
+| `autovacuum_disabled` | (see [`monitorVacuum`](./constructor.md#monitorvacuum)) A job table has `autovacuum_enabled = false`, holds more dead rows than Postgres's own vacuum point, and is still growing with no vacuum running against it at all. Distinct from `xmin_horizon`, where vacuum does run and cannot reclaim: here the fix is autovacuum rather than a stuck transaction, so no holder is named. Stays quiet for an operator vacuuming on their own schedule, whose manual vacuum both moves the timestamp and drops the count. Emitted once per episode | `table`, `tables`, `liveTuples`, `deadTuples`, `budget`, `vacuumAgeSeconds` |
+| `monitor_backoff` | The queue-stats aggregate spent long enough scanning the job table to risk holding autovacuum back, so the next refresh is deferred (see [`monitorVacuum`](./constructor.md#monitorvacuum)). Cached counts are served in the meantime, including to `getQueueStats({ force: true })`; `capturedOn` tells you how old they are. Job expiry and heartbeat failure are unaffected. The fix is to shrink the job table (retention, `deleteAfterSeconds`) or partition its busiest queues | `elapsedSeconds`, `naptimeSeconds`, `backoffSeconds`, `backoffUntil` |
 | `invalid_schedule` | A stored schedule could not be evaluated (for example an unusable `timezone` written by an older release) and was skipped for this cron pass; the remaining schedules are unaffected. Emitted once per broken schedule rather than on every pass, and again if the schedule is edited or the instance restarts | `queue`, `key`, `cron`, `timezone` |
 
 ### Warning Persistence

docs/api/jobs.md

@@ -435,13 +435,13 @@ Returns an array of jobs from a queue
 
     Number of jobs to return
 
-  * `priority`, bool, *default: true*
+  * `priority`, bool — **deprecated, ignored since 12.30.0**
 
-    If true, allow jobs with a higher priority to be fetched before jobs with lower or no priority
+    Jobs are always fetched in priority order. This option existed to skip the priority sort for throughput; the fetch index is now ordered to match the fetch, so there is no sort to skip. Setting it `false` was measured roughly 180x *slower* than leaving it alone, because no index leads with `created_on`. Emits a Node `DeprecationWarning` (code `PGBOSS_DEP_FETCH_SORT`) once per option per instance — run with `--trace-deprecation` to find the call site — and will be rejected in the next major.
 
-  * `orderByCreatedOn`, bool, *default: true*
+  * `orderByCreatedOn`, bool — **deprecated, ignored since 12.30.0**
 
-    If true, jobs are fetched in the order they were created. Set to false to disable this sorting for improved performance when order doesn't matter.
+    Jobs are always fetched in creation order. Same reasoning: the fetch index now provides that order directly, so disabling it saved nothing measurable.
 
   * `includeMetadata`, bool, *default: false*
 

docs/api/ops.md

@@ -120,7 +120,7 @@ Use this where pg-boss cannot run the rebuild itself — the connected role does
 ```js
 const commands = await boss.getReindexCommands()
 // [
-//   'REINDEX INDEX CONCURRENTLY pgboss."job_common_i5"',
+//   'REINDEX INDEX CONCURRENTLY pgboss."job_common_i11"',
 //   'REINDEX INDEX CONCURRENTLY pgboss."job_common_pkey"'
 // ]
 ```

docs/api/workers.md

@@ -51,11 +51,11 @@ The default options for `work()` is 1 job every 2 seconds.
   - **Throwing** from the handler still fails the entire batch, exactly as without `perJobResults`. Use the returned array to express per-job failures; reserve throwing for batch-wide errors.
   - Resolving with anything other than an array is treated as a contract violation and fails the whole batch.
 
-* **priority**, bool, *(default=true)*
+* **priority**, bool — **deprecated, ignored since 12.30.0**
 
   Same as in [`fetch()`](./jobs#fetchname-options)
 
-* **orderByCreatedOn**, bool, *(default=true)*
+* **orderByCreatedOn**, bool — **deprecated, ignored since 12.30.0**
 
   Same as in [`fetch()`](./jobs#fetchname-options)
 

docs/dashboard.md

@@ -194,10 +194,18 @@ const boss = new PgBoss({
 });
 ```
 
-Warnings correlate to `warning` events already emitted by pg-boss:
+Warnings correlate to [`warning`](./api/events.md#warning) events already emitted by pg-boss:
 - `slow_query`: Queries taking longer than expected
 - `queue_backlog`: Queues exceeding their warning threshold
 - `clock_skew`: Database clock drift detection
+- `listen_notify_unavailable`: `useListenNotify` is on but no listener could be established
+- `invalid_schedule`: A stored schedule could not be evaluated and was skipped
+- `index_bloat`: A job index holds far more pages than its live entries need and was not rebuilt
+- `xmin_horizon`: Something is pinning the MVCC horizon, so vacuum reclaims nothing
+- `autovacuum_disabled`: Nothing is vacuuming a job table at all
+- `monitor_backoff`: The queue-stats aggregate was deferred to keep autovacuum moving
+
+A warning type written by a newer pg-boss core than the dashboard is displayed under its raw name.
 
 ## Tech Stack