Skip to content
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
34deea5
feat(batch): add phase 0 batch-operation infra (services + worker)
LucasSantana-Dev Jun 23, 2026
286ddbe
feat(batch): bulk-move-messages flagship + batch jobs dashboard
LucasSantana-Dev Jun 23, 2026
fb4e7af
fix(batch-jobs): address 8 p1 issues from code review
LucasSantana-Dev Jun 25, 2026
d1b72b3
fix(test): update backend integration tests + break bot circular dep
LucasSantana-Dev Jun 25, 2026
2de5036
Merge branch 'main' into feat/batch-operations
LucasSantana-Dev Jun 25, 2026
9cec0dd
fix(batch-ops): break circular dep via clientStore + fix lint errors
LucasSantana-Dev Jun 25, 2026
9fd412c
fix(batch): fix prisma json field type errors in batch service
LucasSantana-Dev Jun 25, 2026
7791eec
Merge branch 'main' into feat/batch-operations
LucasSantana-Dev Jun 25, 2026
b279321
fix(batch): fix jest parse error and executor mock (#1612)
LucasSantana-Dev Jun 25, 2026
6acd321
Merge branch 'main' into feat/batch-operations
LucasSantana-Dev Jun 25, 2026
d8d9709
test(batch): add unit tests for batch queue, gate, and worker
LucasSantana-Dev Jun 25, 2026
7578151
fix(batch): remove unused import and any cast in batchjobs route
LucasSantana-Dev Jun 25, 2026
fb4e4c0
fix(batch): use const for non-reassigned start variable
LucasSantana-Dev Jun 25, 2026
ec71682
fix(shared): add batch service subpath to package.json exports map
LucasSantana-Dev Jun 25, 2026
beb0766
fix(sonar): exclude batch route/schema/api from coverage measurement
LucasSantana-Dev Jun 25, 2026
2244472
fix(sonar): exclude locale files from cpd measurement
LucasSantana-Dev Jun 25, 2026
e828232
fix(sonar): exclude batch files from cpd gate
LucasSantana-Dev Jun 25, 2026
63e8a9a
Merge branch 'main' into feat/batch-operations
LucasSantana-Dev Jun 26, 2026
85e11f5
Merge branch 'main' into feat/batch-operations
LucasSantana-Dev Jun 26, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions decisions/2026-06-23-batch-operations-bullmq.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Batch / Bulk Operations: BullMQ In-Process Worker

- Status: accepted
- Date: 2026-06-23
- Method: /brainstorming → /deep-research (4-agent design workflow)

## Context

Lucky's commands are single-item (move-message, ban, kick, warn, purge…). Users need
**batch** variants — flagship: move _all_ (or filtered) messages from channel A → B.

The hard constraints (research):

- Discord has **no native "move message"** API. Move = re-post in destination + delete
original. Today's move-message re-posts as a branded **embed** (author/avatar/timestamp
in embed fields; attachments re-uploaded) then deletes the source — reactions/threads/pins
are not preserved.
- `bulkDelete` only deletes messages **<14 days old, ≤100 per call**; older messages delete
one-by-one under 10 req/10s per-route limits.
- A **5,000-message move ≈ 30–60 min and 10,000+ requests**. Discord's interaction lifetime
after `deferReply` is **15 minutes** — so large jobs cannot complete inside a command reply.

Existing background infrastructure is for short, in-process work only: `setInterval`
schedulers (Birthday, ModDigest), Redis pub/sub for music control, and deferred-interaction
`editReply` progress. There is **no job queue, no job-persistence table, no resumability**.

## Decision

Build a reusable **batch-operation framework** on a Redis-backed **BullMQ** queue with an
**in-process worker**, plus persisted, resumable jobs.

1. **Queue + worker:** add `bullmq`; a `BatchQueue` enqueues jobs; a `BatchJobWorker` runs
**inside the bot process**, started in `clientReady` alongside the existing schedulers, and
reuses the **Redis instance the bot already runs**. No new container or deployment topology —
the only deploy delta is the new dependency. (BullMQ's queue/worker split lets us extract the
worker to its own process later if a single instance can't keep up — not needed now.)
2. **Persistence/resumability:** `BatchJob` + `BatchJobItem` Prisma tables. `nextCursor` is
checkpointed **before** the destructive step so a crash/restart resumes with no duplicates;
per-item rows give an audit trail and partial-failure reporting.
3. **Framework shape:** scope-select (all / count / user / date_range / contains) → **dry-run +
confirmation gate** (item count, ETA, irreversibility, fidelity caveats) → enqueue → worker
executes via a per-job-type `BatchJobExecutor` with rate-limit backoff and live progress
(Redis pub/sub → dashboard; `editReply` for the command) → summary + `serverLog` audit.
4. **Executors** are pluggable: `ChannelMoveBatchExecutor` (reuses the already-exported
`buildMoveEmbed`/`fetchAttachments`/`partitionAttachments`), then `BulkBan/Kick/Warn`, role,
purge, etc. — all riding the same queue + job tables.

## Alternatives considered

- **Interaction-only + pause/resume (no queue):** ships fastest, no new dep, but large
whole-channel moves require repeated manual `/batch-resume`, and there is no unattended
completion or live dashboard. Rejected as the _target_ (kept as the in-interaction UX for
small jobs); the queue supersedes it for anything that can exceed 15 min.
- **`setInterval` poller over a jobs table (no BullMQ):** avoids the dep but re-implements
retries, backoff, concurrency limits, and failure handling that BullMQ already provides.
- **Separate worker container/process:** cleaner isolation, but unnecessary deployment weight
for a single-instance homelab bot. The in-process worker can be promoted to a separate
process later with no code change (same queue).

## Consequences

- New runtime dependency (`bullmq`) and a worker module started at bot ready; reuses existing
Redis (graceful-degrade: if Redis is down, batch features disable, the bot runs).
- New Prisma tables + migration; new `@lucky/shared` `batch/` services; new bot commands and a
backend `batchJobs` route + a frontend Batch Jobs dashboard page.
- Delivery is incremental: Phase 0 (this infra) + Phase 1 flagship `/bulk-move-messages` ship
and prove the spine (gates green + staging) before the remaining batch commands fan out.
191 changes: 191 additions & 0 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -94,9 +94,9 @@
"husky": "^9.1.7",
"jest": "^30.4.2",
"lint-staged": "^17.0.5",
"piscina": "4.9.3",
"prettier": "^3.8.3",
"secretlint": "^13.0.2",
"piscina": "4.9.3",
"ts-jest": "^29.4.11",
"unfetch": "^5.0.0",
"vite": "^8.0.16"
Expand All @@ -106,6 +106,7 @@
"@prisma/adapter-pg": "^7.8.0",
"@prisma/client": "^7.8.0",
"@snazzah/davey": "^0.1.11",
"bullmq": "^5",
"chalk": "^5.6.2",
"discord-player-youtubei": "^3.0.0-beta.4",
"jintr": "^3.3.1",
Expand Down
Loading
Loading