feat: scalable bots

.env.public

@@ -29,3 +29,12 @@ FRONTEND_URL_PROD=https://canary.automoderator.app
 # #270 - shared threshold (ms) for both Postgres's own log_min_duration_statement (docker-compose.yml)
 # and the app-level createDb() slow-query log (packages/private/db).
 POSTGRES_SLOW_QUERY_LOG_MS=200
+
+# Horizontal scaling. How many gateway shards one replica of each bot should aim to run -- the only number
+# a human sets. Discord's /gateway/bot recommendation decides the shard count, `./compose up` derives the
+# replica count from these two, and each replica claims its own slice against redis. Blank (the default)
+# means one replica running every shard, which is where all three bots are today. Raising a bot above one
+# shard is what makes these meaningful; see docs/roadmap/12-horizontal-scaling.md before setting one.
+AMA_SHARDS_PER_REPLICA=
+MODMAIL_SHARDS_PER_REPLICA=
+SOCIAL_SHARDS_PER_REPLICA=

CLAUDE.md

@@ -14,6 +14,8 @@ Full doc set:
 - [docs/adr/0002-db-stack.md](docs/adr/0002-db-stack.md) — why the DB stack was replaced (implemented, M1).
 - [docs/roadmap/05-migration-cutover.md](docs/roadmap/05-migration-cutover.md) — M4, AMA drain-and-swap cutover (in progress).
 - [docs/roadmap/06-modmail-port.md](docs/roadmap/06-modmail-port.md) — M5, ModMail: feature work shipped, only the legacy data migration + cutover remain.
+- [docs/roadmap/11-automoderator-port.md](docs/roadmap/11-automoderator-port.md) — AutoModerator rebuilt as a monolith on the v3 stack (planned). Supersedes the "AutoModerator is out of scope" framing below and in 00-overview.
+- [docs/roadmap/12-horizontal-scaling.md](docs/roadmap/12-horizontal-scaling.md) — how bots run as N replicas (implemented, off by default). Read before touching `bot-core`'s gateway/session/replica code or adding a DB-driven timer to a bot.
 - [docs/workflow.md](docs/workflow.md) — branching, commits, local dev, verification standard.
 - [docs/frontend.md](docs/frontend.md) — `apps/website` conventions: theme tokens, component library, forms, data fetching. Read before writing UI code.
 

compose

@@ -45,7 +45,91 @@ then
   export GRAFANA_OAUTH_ROLE_ATTRIBUTE_PATH="contains([${_quoted_ids}],id)&&'Admin'"
 fi
 
+# Replica counts for sharded bots are computed here rather than written down anywhere, so the only
+# number a human maintains is <BOT>_SHARDS_PER_REPLICA. Discord's own /gateway/bot recommendation
+# decides the shard count; each replica then works out which slice is its own against redis (see
+# packages/private/bot-core/src/lib/replica.ts).
+#
+# Host-side on purpose: this box already has docker access, so nothing needs the docker socket
+# mounted into a container that processes untrusted Discord input.
+# Same precedence as the merged env file above (.env.private wins), and the same grep/cut approach as
+# the ADMINS block: read the value without ever letting the file's contents be evaluated as shell.
+read_env() {
+  local key="$1" public_value='' private_value=''
+
+  if [ -f .env.public ]
+  then
+    public_value="$(grep -m1 "^${key}=" .env.public | cut -d= -f2-)"
+  fi
+
+  if [ -f .env.private ]
+  then
+    private_value="$(grep -m1 "^${key}=" .env.private | cut -d= -f2-)"
+  fi
+
+  printf '%s' "${private_value:-$public_value}"
+}
+
+SCALE_ARGS=()
+plan_scale() {
+  local service="$1" token_key="$2" shards_per_replica_key="$3"
+  local shards_per_replica token response shards replicas running
+
+  shards_per_replica="$(read_env "$shards_per_replica_key")"
+  # Unset means this bot isn't sharded across replicas, which is the default -- leave compose alone.
+  [ -n "$shards_per_replica" ] || return 0
+  if ! [[ "$shards_per_replica" =~ ^[0-9]+$ ]] || [ "$shards_per_replica" -lt 1 ]
+  then
+    echo "${shards_per_replica_key} must be a positive integer, got: ${shards_per_replica}" >&2
+    exit 1
+  fi
+
+  token="$(read_env "$token_key")"
+  if [ -z "$token" ]
+  then
+    echo "warning: ${shards_per_replica_key} is set but ${token_key} is empty; leaving ${service} scale untouched" >&2
+    return 0
+  fi
+
+  response="$(curl -sS --max-time 10 -H "Authorization: Bot ${token}" \
+    https://discord.com/api/v10/gateway/bot 2>/dev/null)" || response=''
+  shards="$(printf '%s' "$response" | grep -o '"shards"[[:space:]]*:[[:space:]]*[0-9]\+' | grep -o '[0-9]\+$')"
+
+  if ! [[ "$shards" =~ ^[0-9]+$ ]] || [ "$shards" -lt 1 ]
+  then
+    # Deliberately keeps whatever is already running instead of falling back to 1: silently scaling a
+    # sharded bot down to a single replica because Discord was briefly unreachable would be a far worse
+    # outcome than deploying with a stale-but-correct replica count.
+    running="$(docker compose -f docker-compose.yml "${ENV_FILE_ARGS[@]}" ps -q "$service" 2>/dev/null | grep -c .)"
+    if [ "${running:-0}" -gt 0 ]
+    then
+      echo "warning: could not read Discord's shard count for ${service}; holding at ${running} replica(s)" >&2
+      SCALE_ARGS+=(--scale "${service}=${running}")
+    else
+      echo "warning: could not read Discord's shard count for ${service}; leaving its scale untouched" >&2
+    fi
+
+    return 0
+  fi
+
+  replicas=$(( (shards + shards_per_replica - 1) / shards_per_replica ))
+  echo "${service}: ${shards} shard(s) / ${shards_per_replica} per replica -> ${replicas} replica(s)"
+  SCALE_ARGS+=(--scale "${service}=${replicas}")
+}
+
+# Only for `up`. Every other subcommand (logs, stop, ps, exec, ...) must not silently restructure the
+# deployment, and `--scale` isn't meaningful for them anyway.
+if [ "$1" = 'up' ]
+then
+  plan_scale ama-bot AMA_BOT_TOKEN AMA_SHARDS_PER_REPLICA
+  # Only the public deployment: a custom instance (#216) is single-guild by definition, so it is always
+  # one shard and one replica.
+  plan_scale modmail-bot MODMAIL_BOT_TOKEN MODMAIL_SHARDS_PER_REPLICA
+  plan_scale social-bot SOCIAL_BOT_TOKEN SOCIAL_SHARDS_PER_REPLICA
+fi
+
 docker compose \
   -f docker-compose.yml \
   "${ENV_FILE_ARGS[@]}" \
-  "$@"
+  "$@" \
+  "${SCALE_ARGS[@]}"

docker-compose.yml

@@ -233,6 +233,11 @@ services:
     depends_on:
       discord-proxy:
         condition: service_healthy
+    # Maps the per-bot value from the env files onto the generic name bot-core reads, so `./compose` can
+    # size this service's replica count from it while the container itself stays bot-agnostic. Unset means
+    # one replica running every shard. `./compose up` computes `--scale` from the same value.
+    environment:
+      SHARDS_PER_REPLICA: ${AMA_SHARDS_PER_REPLICA:-}
     command: ['node', '--enable-source-maps', './services/ama-bot/dist/bin.js']
     volumes:
       - ./logs/ama-bot:/usr/chatsift/logs/ama-bot
@@ -249,6 +254,8 @@ services:
     depends_on:
       discord-proxy:
         condition: service_healthy
+    environment:
+      SHARDS_PER_REPLICA: ${MODMAIL_SHARDS_PER_REPLICA:-}
     command: ['node', '--enable-source-maps', './services/modmail-bot/dist/bin.js']
     volumes:
       - ./logs/modmail-bot:/usr/chatsift/logs/modmail-bot
@@ -265,6 +272,8 @@ services:
     depends_on:
       discord-proxy:
         condition: service_healthy
+    environment:
+      SHARDS_PER_REPLICA: ${SOCIAL_SHARDS_PER_REPLICA:-}
     command: ['node', '--enable-source-maps', './services/social-bot/dist/bin.js']
     volumes:
       - ./logs/social-bot:/usr/chatsift/logs/social-bot

docs/roadmap/11-automoderator-port.md

@@ -29,9 +29,9 @@ Recorded so they don't get re-litigated.
    has a mechanism in `backend-core`.
 3. **Per-feature phasing.** Foundations first (schema baseline, API surface, dashboard scaffold, observability,
    dev affordances), then one feature at a time through the full stack.
-4. **First bot to opt into bot-core horizontal scaling.** The mechanism doesn't exist yet; this doc's job is to
-   ensure AutoModerator is _shaped_ to opt in later as a config change, not a rewrite. See
-   [Scaling readiness](#scaling-readiness).
+4. **First bot to opt into bot-core horizontal scaling.** The mechanism now exists
+   ([12-horizontal-scaling.md](12-horizontal-scaling.md)) and opting in is configuration, as intended. This doc's
+   job is still to ensure AutoModerator is _shaped_ for it. See [Scaling readiness](#scaling-readiness).
 5. **The invite worker is dropped.** `invite-lookup.chatsift.workers.dev` is live but its source isn't in this repo
    and nothing on `main` calls it. Invite resolution happens through the bot's own REST client instead.
 6. **Banword matching is delegated to Discord.** Feature 01 ships no matcher — see
@@ -134,8 +134,9 @@ So the broker disappears with the split that created it. Redis's actual roles he
    `backend-core/src/lib/realtimeBroadcast.ts` over the `ws:invalidate` channel, consumed by the API's WS gateway.
    Every mutating route declares `realtimeChannel` and gets it for free.
 3. **The guild list** — `bot:AUTOMODERATOR`, same as every other bot.
-4. **Distributed locks** — _not needed until P8._ `withGuildUserLock` is process-local, which is correct for one
-   replica. See [Scaling readiness](#scaling-readiness) for what has to become a Redis lock when that changes.
+4. **Distributed locks** — still not needed at P8, in most places. `withGuildUserLock` is process-local, which
+   stays correct under sharding because a guild's events only ever reach one replica. See
+   [Scaling readiness](#scaling-readiness) item 4 for the narrow set that genuinely needs a Redis lock.
 
 **No queue library.** The scheduler stays a polled task table, as it was in legacy. BullMQ or similar would be a new
 dependency buying a retry/backoff/delay model the `tasks` table already implements in ~40 lines, and would put job
@@ -395,8 +396,16 @@ sanity on the message cache, and a pass over every `ActionExecutor` call site co
 
 ### P8 — Horizontal scaling opt-in
 
-**Blocked on bot-core.** Ships when the scaling mechanism exists. This phase is the opt-in plus whatever the
-mechanism requires; the invariants that make it cheap are held from P0 onward and listed below.
+**No longer blocked** — the mechanism shipped, see [12-horizontal-scaling.md](12-horizontal-scaling.md). This
+phase is now genuinely just configuration, provided the invariants below were held from P0:
+
+- Add `AUTOMODERATOR_SHARDS_PER_REPLICA` to `.env.public` and map it onto the service's `SHARDS_PER_REPLICA` in
+  `docker-compose.yml`, following the three existing bot blocks.
+- Add a `plan_scale automoderator-bot AUTOMODERATOR_BOT_TOKEN AUTOMODERATOR_SHARDS_PER_REPLICA` line to `./compose`.
+- Audit every DB-driven timer this port adds for `ownsShardForGuild`. The scheduler does not need it (`SKIP LOCKED`
+  already claims), but anything that reads rows and then acts on Discord does.
+
+The bot itself needs no code change to opt in: `createBotGateway` claims a slot on every boot regardless.
 
 ---
 
@@ -434,8 +443,15 @@ than a rewrite.
 3. **The scheduler claims rather than reads.** `FOR UPDATE SKIP LOCKED` from P2, so N replicas is safe by
    construction rather than by leader election.
 4. **Read-modify-write paths are enumerated and lock-ready.** Ladder counting, report dedupe and case-number
-   allocation are the three. Case numbers are already database-allocated; the other two need a Redis lock the day
-   replica count exceeds one — flagged in code at their call sites, not just here.
+   allocation are the three. Case numbers are already database-allocated.
+
+   **Narrowed once the mechanism landed** ([12-horizontal-scaling.md](12-horizontal-scaling.md)): the other two do
+   _not_ automatically need a Redis lock. A guild maps to exactly one shard owned by exactly one replica, so
+   `withGuildUserLock` still serializes every guild-scoped gateway event and interaction for a guild+user pair,
+   exactly as it does today — and DMs always arrive on shard 0. What genuinely needs a distributed lock is
+   narrower: state `services/api` also mutates, and anything keyed on something other than a guild. Check which
+   category a call site is in rather than assuming the broad version.
+
 5. **Metrics are replica-safe.** No metric assumes a single process; the scrape config gains per-replica targets
    rather than the code aggregating.