Skip to content

feat(jobs): add state, limit, ordering and cursor filters to findJobs() - #889

Draft
kibertoad wants to merge 1 commit into
timgit:masterfrom
kibertoad:feat/find-jobs-filters
Draft

feat(jobs): add state, limit, ordering and cursor filters to findJobs()#889
kibertoad wants to merge 1 commit into
timgit:masterfrom
kibertoad:feat/find-jobs-filters

Conversation

@kibertoad

Copy link
Copy Markdown
Contributor

findJobs() has no bound and no order. Every call returns the queue's entire retained history for the given filters, in whatever order the planner produces. That makes it awkward for the admin and inspection work it exists for:

  • findJobs('email-send', { key: 'user-123' }) on a key that has been busy for a year returns a year of jobs. There is no way to ask for the last twenty.
  • Filtering on state means fetching everything and filtering in JavaScript, or using queued, which only expresses created + retry. "Show me the failed ones" is a full table read.
  • Nothing is orderable, so there is no stable way to walk a large result set.

This adds four options, plus a companion read.

states

await boss.findJobs('email-send', { states: ['failed', 'cancelled'] })

queued is the special case ['created', 'retry']. Supplying both is rejected rather than silently resolved in favour of one: { queued: true, states: ['completed'] } has no reading that a caller would predict.

limit, orderBy, direction

await boss.findJobs('email-send', { limit: 20, direction: 'desc' })

orderBy is createdOn (default) or startAfter. Both are immutable for the life of a job, which is what makes them safe to sort on: a job that changes state mid-pagination keeps its place in the ordering instead of jumping to another page.

cursor

Keyset pagination on (order column, id), not OFFSET:

let cursor

for (;;) {
  const page = await boss.findJobs('email-send', { states: ['failed'], limit: 100, cursor })
  if (page.length === 0) break
  cursor = page[page.length - 1].id
}

The anchor's sort value is read back from the row the cursor names and the row comparison walks strictly past it. A job inserted or deleted between pages cannot shift the window into repeating or skipping a row, and page N does not cost more than page 1. id is the tiebreaker so the order is total: without it, two jobs sharing a created_on could sort either way between calls and land on both sides of a page boundary.

getJobByKey(name, key, options)

The singleton-key counterpart of getJobById(). A key identifies a series rather than a row, so it answers with the most recent job:

// what happened last for this key
const last = await boss.getJobByKey('order-processing', 'order-42')

// what the key currently has outstanding
const pending = await boss.getJobByKey('order-processing', 'order-42', { queued: true })

Compatibility

Ordering is opt-in. A call that supplies none of limit, orderBy, direction, or cursor produces the statement findJobs() has always issued, unordered, with the plan it has today. No existing call changes shape or cost.

One drive-by fix: findJobs() now applies the CockroachDB integer normalization fetch() and getJobById() already perform, so a job read through it has the same shape on every backend. Without it getJobByKey() would return retryLimit as a string where getJobById() returns a number.

Tests

test/findJobsFiltersTest.ts: state filtering including the rejected combinations, bounded results, both orderings, forward and backward paging, a cursor naming a deleted row's successor (the case an offset pager gets wrong), a cursor naming nothing, state filter combined with paging, and the unordered path staying unordered. Plus getJobByKey newest-first, the queued narrowing, key isolation, and the null cases.

findJobs() has no bound and no order. Every result set is the queue's entire
retained history for the given filters, in whatever order the planner
produces, which makes it unusable for the admin and inspection work it is
meant for: a key with a year of history returns a year of history, and there
is no way to ask for the last twenty.

Adds four options that turn it into a paging read:

- states: filter on an explicit list of job states. queued already covers
  created + retry; states covers any subset, including terminal ones.
  Combining the two is rejected rather than silently resolved.
- limit: bound the result.
- orderBy (createdOn | startAfter) and direction: define the order. Both
  columns are immutable for the life of a job, so a job that changes state
  mid-pagination keeps its place.
- cursor: keyset pagination on (order column, id). The anchor's sort value is
  read back from the row the cursor names and the row comparison walks
  strictly past it, so a row inserted or deleted between pages cannot shift
  the window and page N does not cost more than page 1. id breaks ties so the
  order is total.

Ordering stays opt-in: a call that supplies none of limit, orderBy, direction
or cursor gets the statement findJobs has always issued, unordered and with
the plan it has today.

Also adds getJobByKey(name, key, options), the singleton-key counterpart of
getJobById(). A key names a series rather than a row, so it answers with the
most recent job, and { queued: true } narrows it to what the key currently has
outstanding.

findJobs() now applies the CockroachDB integer normalization fetch() and
getJobById() already perform, so a job read through it has the same shape on
every backend.
@coveralls

Copy link
Copy Markdown

Coverage Status

coverage: 99.894% (-0.1%) from 100.0% — kibertoad:feat/find-jobs-filters into timgit:master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants