Skip to content

Commit c8c99b9

Browse files
committed
chore: last coderabbit nits
1 parent a9ca42f commit c8c99b9

2 files changed

Lines changed: 44 additions & 12 deletions

File tree

docs/roadmap/12-horizontal-scaling.md

Lines changed: 35 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -61,9 +61,20 @@ when the cluster is at its intended size.
6161
Two properties fall out of this rather than needing rules of their own:
6262

6363
- **Claims are atomic, so two replicas can never hold the same index.** The settle window is therefore never a
64-
correctness concern, however the timing falls.
65-
- **Coverage is always complete.** An index nobody claimed is picked up by the replica below it, so running fewer
66-
replicas than intended means somebody works harder, not that a guild stops being watched.
64+
correctness concern, however the timing falls. This one holds at every instant.
65+
- **Coverage converges.** An index nobody claimed gets picked up by the replica below it, so running fewer
66+
replicas than intended costs effort rather than coverage — but that is the _settled_ state, not a continuous
67+
guarantee. A replica that dies takes its shards down until something reclaims its index, so **transient dark
68+
shards are possible and expected**:
69+
70+
| How the index came free | Dark for |
71+
| ---------------------------------------- | ------------------------------------------------------------------------------------------------ |
72+
| Crash or planned restart | that container's restart, typically seconds — it reclaims its own index |
73+
| Removed for good (scale-down, host gone) | up to the lease TTL, plus two watcher checks, plus the neighbour's restart — on the order of 90s |
74+
75+
Nothing shortens the second case without giving up the debounce that stops a peer merely between renewals from
76+
bouncing the cluster. If a bot ever needs sub-minute recovery from a permanently removed replica, that is a
77+
reason to revisit the watcher interval, not the assignment scheme.
6778

6879
An index owns a slice that is the same shards no matter how many peers are up. That is what lets a replica hold
6980
its assignment for its whole lifetime instead of resharding underneath itself whenever the cluster changes size.
@@ -77,17 +88,17 @@ instead of `4/4/4/2` for 14 shards) and was rejected anyway, because it trades t
7788
- **An index must mean the same shards regardless of the current shard count.** Flat slices survive Discord
7889
raising its recommendation: 14 → 15 grows only the tail replica. Proportional boundaries move on almost any
7990
bump, so every replica reshards, every replica restarts, and every replica's guilds change.
80-
- **`SHARDS_PER_REPLICA` stays the capacity bound its name promises**, not a divisor. A replica never exceeds it
81-
unless it is covering for a peer that never claimed its index.
91+
- **`SHARDS_PER_REPLICA` stays a shard count**, not a divisor — index `i` means the same slice whatever the
92+
cluster is doing, which is what makes the previous point true.
8293

8394
So a short tail is **headroom, not imbalance**`4/4/4/2` fills to `4/4/4/3`, `4/4/4/4`, and then a fifth index
84-
appears. Sizing a container is a question about `SHARDS_PER_REPLICA` alone.
95+
appears.
8596

8697
Genuine imbalance has exactly two causes, and both are the cluster not being at its intended size:
8798

8899
- **A replica is missing.** A peer covers its indices. This is unavoidable rather than a design choice: if three
89-
replicas must cover four indices' worth of shards, one of them holds more. The alternative is dark shards, which
90-
is the trade this design deliberately refuses.
100+
replicas must cover four indices' worth of shards, one of them holds more. The alternative is leaving those
101+
shards uncovered for as long as the replica stays away, which is the trade this design refuses.
91102
- **A straggler.** A replica starting after its peers' settle window finds everything claimed and idles as a hot
92103
spare while some peer holds two indices. See below.
93104

@@ -97,6 +108,22 @@ appearing or disappearing reshards (and therefore restarts) the entire cluster,
97108
even a moment about the live count produces overlapping or missing shards. That is a materially worse failure mode
98109
than one replica temporarily carrying an extra index.
99110

111+
### Sizing a container
112+
113+
`SHARDS_PER_REPLICA` is the **steady-state target, not a hard cap**, so it is not the only input to sizing. A
114+
replica holds the union of every index it claims, and it claims extras precisely when peers are missing — which is
115+
also when the surviving replicas are carrying the most load:
116+
117+
| Cluster state | Shards on the heaviest replica |
118+
| ------------------------------ | ------------------------------ |
119+
| Fully provisioned | `SHARDS_PER_REPLICA` |
120+
| One peer missing | `2 × SHARDS_PER_REPLICA` |
121+
| Worst case (only one survivor) | the entire `shardCount` |
122+
123+
Size for at least **twice** `SHARDS_PER_REPLICA` if a single replica loss should be absorbed without degrading,
124+
and treat `shardsOwned` in the boot log (and the `covering for missing replicas` message) as the signal that a
125+
replica is running above its target.
126+
100127
### Changing shards means restarting
101128

102129
`@discordjs/ws` cannot add shards to a live `WebSocketManager``updateShardCount` tears everything down and

packages/private/bot-core/src/lib/replica.ts

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -137,10 +137,15 @@ export interface ReplicaSlot {
137137
* so peers starting alongside can claim theirs, then greedily claim whatever is still free immediately above.
138138
*
139139
* Complete coverage falls out of that greedy step rather than needing a rule of its own -- an index left
140-
* unclaimed by a peer that never started is picked up by the replica below it, so the union of every replica's
141-
* shards is always the full set. The cost of running too few replicas is therefore that somebody works harder,
142-
* never that a guild stops being watched. Running too many is equally safe: the surplus finds nothing free and
143-
* waits as a hot spare.
140+
* unclaimed by a peer that never started is picked up by the replica below it, so once the cluster settles the
141+
* union of every replica's shards is the full set. The cost of running too few replicas is therefore that
142+
* somebody works harder, not that a guild goes unwatched. Running too many is equally safe: the surplus finds
143+
* nothing free and waits as a hot spare.
144+
*
145+
* That is the settled state, not a continuous guarantee: a replica that dies takes its shards down until
146+
* something reclaims its index -- its own restart in the ordinary case, or `startWatching` promoting a neighbour
147+
* when it is gone for good. Transient dark shards are expected; see docs/roadmap/12-horizontal-scaling.md for how
148+
* long each case lasts.
144149
*
145150
* The known wart is a *straggler*: a replica that starts well after `settleMs` has passed for its peers finds
146151
* everything already claimed and idles as a hot spare, leaving the cluster correct but unbalanced until the next

0 commit comments

Comments
 (0)