fixes and tests

Author: timgit

docs/api/constructor.md

@@ -149,6 +149,7 @@ The following configuration options should not normally need to be changed, but
   | --- | --- | --- | --- |
   | `minPages` | int | 128 | Ignore indexes smaller than this many 8 kB pages |
   | `maxEntriesPerPage` | number | 5 | Live entries per page below which an index counts as bloated. A freshly built job index holds 140-170 |
+  | `minSizeRatio` | number | 4 | How many times larger than its live entries need an index must be. The needed size is estimated from `pg_stats`, so a wide `singletonKey` — which legitimately packs fewer than five entries per page — is not mistaken for bloat |
   | `maxIndexBytes` | int | 2147483648 | Never rebuild an index larger than this |
 
   ```js

docs/api/ops.md

@@ -81,6 +81,8 @@ const version = await boss.schemaVersion()
 
 Runs one maintenance pass immediately instead of waiting for the next background cycle: monitoring (backlog warnings, expired and heartbeat-abandoned jobs, cached stats), deletion of jobs past their retention, warning and queue-stat pruning, and the index bloat check.
 
+Passing `name` restricts the pass to that queue's own rows, but the index bloat check works on tables: for a queue with `partition: false` (the default) the indexes it would rebuild belong to the shared `job_common` table, which every other unpartitioned queue also uses.
+
 This is the same pass the background supervisor runs on `superviseIntervalSeconds`. Call it directly when you have set `supervise: false` and drive maintenance yourself, or in tests where waiting for a timer is not an option.
 
 ```js
@@ -109,11 +111,11 @@ Steps within a pass are individually rate-limited by their own intervals (`maint
 ### `getReindexCommands(options)`
 
 **Arguments**
-- `options`: object, optional. Accepts `force`, `minPages`, `maxEntriesPerPage`, and `maxIndexBytes`.
+- `options`: object, optional. Accepts `force`, `minPages`, `maxEntriesPerPage`, `minSizeRatio`, and `maxIndexBytes`.
 
 Returns the SQL statements that would rebuild the currently bloated job indexes, in the order they should be run, including a `DROP INDEX CONCURRENTLY` for any invalid stub left behind by an interrupted rebuild.
 
-Use this where pg-boss cannot run the rebuild itself — the connected role does not own the indexes, or the `db` adapter wraps queries in a transaction (`REINDEX CONCURRENTLY` cannot run inside one). Unlike the background pass, no ownership filter and no size cap are applied unless `maxIndexBytes` is passed, since the commands are intended for an operator who may run them as a different role.
+Use this where pg-boss cannot run the rebuild itself — the connected role does not own the indexes, or the `db` adapter wraps queries in a transaction (`REINDEX CONCURRENTLY` cannot run inside one). Returns an empty array on CockroachDB and YugabyteDB, which have no btree bloat to reclaim and reject `REINDEX` in any form. Unlike the background pass, no ownership filter and no size cap are applied unless `maxIndexBytes` is passed, since the commands are intended for an operator who may run them as a different role.
 
 ```js
 const commands = await boss.getReindexCommands()

docs/api/utils.md

@@ -50,9 +50,9 @@ const sql = getRollbackPlans('pgboss', 36)
 
 **Arguments**
 - `schema`: string, database schema name
-- `options`: object, optional. Accepts `minPages` (default 128) and `maxEntriesPerPage` (default 5).
+- `options`: object, optional. Accepts `minPages` (default 128), `maxEntriesPerPage` (default 5) and `minSizeRatio` (default 4).
 
-Returns the catalog query pg-boss uses to find bloated job indexes, as SQL text. Unlike [`getReindexCommands()`](./ops.md#getreindexcommandsoptions) this needs no instance and no connection from this process — it is meant to be pasted into psql or handed to a monitoring tool.
+Returns the catalog query pg-boss uses to find bloated job indexes, as SQL text. PostgreSQL only — CockroachDB and YugabyteDB do not answer it. Unlike [`getReindexCommands()`](./ops.md#getreindexcommandsoptions) this needs no instance and no connection from this process — it is meant to be pasted into psql or handed to a monitoring tool.
 
 ```js
 const sql = getIndexBloatPlans('pgboss')

src/attorney.ts

@@ -738,12 +738,14 @@ function validateReindexConfig (config: any) {
     'configuration assert: reindex must be a boolean or an options object')
 
   if (typeof config.reindex === 'object' && config.reindex !== null) {
-    const { minPages, maxEntriesPerPage, maxIndexBytes, force } = config.reindex
+    const { minPages, maxEntriesPerPage, minSizeRatio, maxIndexBytes, force } = config.reindex
 
     assert(minPages === undefined || (Number.isInteger(minPages) && minPages >= 0),
       'configuration assert: reindex.minPages must be an integer >= 0')
     assert(maxEntriesPerPage === undefined || (typeof maxEntriesPerPage === 'number' && maxEntriesPerPage > 0),
       'configuration assert: reindex.maxEntriesPerPage must be a number > 0')
+    assert(minSizeRatio === undefined || (typeof minSizeRatio === 'number' && minSizeRatio >= 0),
+      'configuration assert: reindex.minSizeRatio must be a number >= 0')
     assert(maxIndexBytes === undefined || (Number.isInteger(maxIndexBytes) && maxIndexBytes > 0),
       'configuration assert: reindex.maxIndexBytes must be an integer > 0')
     // force belongs to an explicit supervise() call, not to a background timer that would then

src/boss.ts

@@ -44,6 +44,9 @@ class Boss extends EventEmitter implements types.EventsMixin {
   // Warn once per bloated index rather than on every pass. An index leaves the set as soon as it
   // stops qualifying — rebuilt, dropped, or refilled — so a later episode warns again.
   #warnedBloat = new Set<string>()
+  // Local rate limit for passes that only report bloat, which deliberately leave the shared interval
+  // claim to whichever instance can act on it.
+  #detectOnly = 0
 
   events = events
 
@@ -344,13 +347,24 @@ class Boss extends EventEmitter implements types.EventsMixin {
 
     const resolved = this.#resolveReindexOptions(options)
     const force = !!resolved?.force
+    const rebuilding = resolved !== null && !this.#reindexUnavailable
 
-    // An explicit force is a request to run now; everything else waits for the shared interval so
-    // exactly one instance in the cluster does the work.
+    // An explicit force is a request to run now; everything else waits for an interval.
+    //
+    // Which interval depends on whether this instance can do the work. The shared claim exists so
+    // exactly one instance in the cluster rebuilds per window — an instance that is only ever going
+    // to report bloat has no business taking it, or a peer configured to rebuild would find the
+    // window gone and skip the rebuild for a whole day. Detection-only passes throttle themselves
+    // locally instead, on the same interval.
     if (!force) {
-      const claim = plans.trySetReindexTime(this.#config.schema, this.#config.reindexIntervalSeconds)
-      const { rows } = await this.#executeQuery(claim)
-      if (!rows.length) return
+      if (rebuilding) {
+        const claim = plans.trySetReindexTime(this.#config.schema, this.#config.reindexIntervalSeconds)
+        const { rows } = await this.#executeQuery(claim)
+        if (!rows.length) return
+      } else {
+        if (Date.now() < this.#detectOnly) return
+        this.#detectOnly = Date.now() + this.#config.reindexIntervalSeconds * 1000
+      }
     }
 
     if (this.#stopping) return
@@ -359,8 +373,6 @@ class Boss extends EventEmitter implements types.EventsMixin {
     const detectSql = plans.getBloatedIndexes(this.#config.schema, scope, resolved ?? undefined)
     const { rows: bloated } = await this.#executeQuery(detectSql)
 
-    const rebuilding = resolved !== null && !this.#reindexUnavailable
-
     let targets: types.IndexBloat[] = []
 
     if (rebuilding) {
@@ -407,7 +419,7 @@ class Boss extends EventEmitter implements types.EventsMixin {
   // must not stop the rebuilds, since the retry's own IF EXISTS handles the common case anyway.
   async #dropReindexLeftovers (tables?: string[]) {
     try {
-      const { rows } = await this.#executeQuery(plans.getReindexLeftovers(this.#config.schema, tables))
+      const { rows } = await this.#executeQuery(plans.getReindexLeftovers(this.#config.schema, tables, this.#config.noIndexProgressView))
 
       for (const leftover of rows) {
         if (this.#stopping) return
@@ -435,12 +447,17 @@ class Boss extends EventEmitter implements types.EventsMixin {
 
       if (this.#warnedBloat.has(index.name)) continue
 
+      // Ownership first: an index the role cannot touch was never a candidate, so it has no entry in
+      // `failed` no matter why the pass stopped. #reindexUnavailable comes next and covers the
+      // indexes the 25001 giveup skipped without attempting — they are neither failed nor rebuilt,
+      // and reporting a size cap they are nowhere near would point at the wrong knob.
       const reason = failed.get(index.name) ??
-        (!rebuilding
-          ? (this.#reindexUnavailable ?? 'automatic reindexing is disabled')
-          : !index.owned
-              ? 'the connected role does not own the index'
-              : 'the index is larger than maxIndexBytes')
+        (!index.owned
+          ? 'the connected role does not own the index'
+          : this.#reindexUnavailable ??
+            (!rebuilding
+              ? 'automatic reindexing is disabled'
+              : 'the index is larger than maxIndexBytes'))
 
       await emitAndPersistWarning(this.#warningContext,
         WARNING_TYPES.INDEX_BLOAT,
@@ -458,6 +475,11 @@ class Boss extends EventEmitter implements types.EventsMixin {
    * is passed — the commands are for an operator, who may run them as a different role.
    */
   async getReindexCommands (options?: types.ReindexOptions): Promise<string[]> {
+    // The catalog query reads pg_class.relpages and pg_relation_size(), which the heap-less engines
+    // either reject outright or answer with zeroes — same gate as #reindex, and there is nothing to
+    // rebuild on them anyway.
+    if (this.#config.noReindex) return []
+
     const schema = this.#config.schema
 
     const sql = options?.force
@@ -472,21 +494,9 @@ class Boss extends EventEmitter implements types.EventsMixin {
 
     if (!targets.length) return []
 
-    const { rows: leftovers } = await this.#executeQuery(plans.getReindexLeftovers(schema))
-    const commands: string[] = []
-
-    for (const target of targets) {
-      // Postgres names the transient index `<index>_ccnew`, then `_ccnew1`, `_ccnew2` on collision.
-      for (const leftover of leftovers) {
-        if (leftover.name.startsWith(`${target.name}_ccnew`)) {
-          commands.push(plans.dropIndexConcurrently(schema, leftover.name))
-        }
-      }
-
-      commands.push(plans.reindexIndex(schema, target.name))
-    }
+    const { rows: leftovers } = await this.#executeQuery(plans.getReindexLeftovers(schema, undefined, this.#config.noIndexProgressView, true))
 
-    return commands
+    return plans.buildReindexCommands(schema, targets, leftovers)
   }
 }
 

src/cli.ts

@@ -538,7 +538,22 @@ async function cmdReindex (args: ReturnType<typeof parseCliArgs>): Promise<void>
     }
 
     const sql = args.force ? plans.getJobIndexes(schema) : plans.getBloatedIndexes(schema)
-    const { rows } = await db.executeSql(sql)
+
+    let rows: any[]
+
+    try {
+      ({ rows } = await db.executeSql(sql))
+    } catch (err: any) {
+      // The check reads pg_class.relpages and pg_relation_size(). CockroachDB has neither (it
+      // rejects `reltuples / relpages` as an unsupported binary operator) and YugabyteDB reports
+      // zeroes for every relation. The CLI takes a connection string, not a backend profile, so
+      // there is nothing to gate on ahead of time — say what happened instead of surfacing a raw
+      // catalog error.
+      console.error(`Could not read index statistics from schema "${schema}": ${err.message}`)
+      console.error('The bloat check reads pg_class.relpages and pg_relation_size(), which CockroachDB and YugabyteDB do not provide.')
+      process.exitCode = 1
+      return
+    }
 
     if (!rows.length) {
       console.log(args.force
@@ -547,24 +562,22 @@ async function cmdReindex (args: ReturnType<typeof parseCliArgs>): Promise<void>
       return
     }
 
-    const { rows: leftovers } = await db.executeSql(plans.getReindexLeftovers(schema))
-
-    const commands: string[] = []
-
-    for (const index of rows) {
-      for (const leftover of leftovers) {
-        // Postgres names the transient index `<index>_ccnew`, then `_ccnew1`, `_ccnew2` on collision.
-        if (leftover.name.startsWith(`${index.name}_ccnew`)) {
-          commands.push(plans.dropIndexConcurrently(schema, leftover.name))
-        }
-      }
-
-      commands.push(plans.reindexIndex(schema, index.name))
-    }
+    // anyOwner for the printed list, which may be run as a different role; the execution path below
+    // filters ownership itself.
+    const { rows: leftovers } = await db.executeSql(plans.getReindexLeftovers(schema, undefined, false, args.dryRun))
 
     if (args.dryRun) {
+      // Every index, ownership included: the printed SQL is for an operator, who may well run it as
+      // the role that does own them.
+      const commands = plans.buildReindexCommands(schema, rows, leftovers)
+
       console.log(`-- SQL to rebuild ${rows.length} index(es) in schema "${schema}":`)
       for (const command of commands) console.log(`${command};`)
+
+      for (const index of rows.filter(i => !i.owned)) {
+        console.log(`-- note: "${index.name}" is not owned by ${config.user || 'the connected role'} and needs a role that can REINDEX it`)
+      }
+
       return
     }
 
@@ -573,9 +586,25 @@ async function cmdReindex (args: ReturnType<typeof parseCliArgs>): Promise<void>
       console.log(`  ${index.table}.${index.name} — ${mb} MB across ${index.pages} pages, ~${index.entries} live entries`)
     }
 
-    console.log(`\nRebuilding ${rows.length} index(es)...`)
+    // Executing is different from printing: REINDEX needs ownership, so an index this role cannot
+    // touch is reported up front rather than attempted for the sake of the server's error message.
+    const skipped = rows.filter(index => !index.owned)
+    const targets = rows.filter(index => index.owned)
+
+    for (const index of skipped) {
+      console.error(`  ✗ ${index.name}\n    the connected role does not own this index and cannot reindex it`)
+    }
+
+    if (!targets.length) {
+      process.exitCode = 1
+      return
+    }
+
+    const commands = plans.buildReindexCommands(schema, targets, leftovers)
+
+    console.log(`\nRebuilding ${targets.length} index(es)...`)
 
-    let failures = 0
+    let failures = skipped.length
 
     for (const command of commands) {
       try {

src/index.ts

@@ -35,7 +35,8 @@ export function getMigrationPlans (schema?: string, version?: number, options?:
 
 /**
  * The catalog query pg-boss uses to find bloated job indexes, as SQL text. Runnable in psql with no
- * pg-boss instance and no connection from this process.
+ * pg-boss instance and no connection from this process. PostgreSQL only — the heap-less engines
+ * (CockroachDB, YugabyteDB) do not answer it.
  */
 export function getIndexBloatPlans (schema?: string, options?: types.IndexBloatOptions) {
   return plans.getBloatedIndexes(schema || plans.DEFAULT_SCHEMA, undefined, options)
@@ -468,8 +469,9 @@ export class PgBoss extends EventEmitter<types.PgBossEventMap> {
    * left by an interrupted rebuild.
    *
    * For installations where pg-boss cannot run them itself — a role that doesn't own the indexes,
-   * an adapter that wraps queries in a transaction, or a distributed backend. Pass
-   * `{ force: true }` for every job index rather than only the bloated ones.
+   * or an adapter that wraps queries in a transaction. Pass `{ force: true }` for every job index
+   * rather than only the bloated ones. Empty on CockroachDB and YugabyteDB, which have no btree
+   * bloat to reclaim and reject `REINDEX` in any form.
    */
   getReindexCommands (options?: types.ReindexOptions): Promise<string[]> {
     return this.#boss.getReindexCommands(options)