[serve] Cache per-tick replica id walks behind a container mutation version by johntaylor-cell · Pull Request #64910 · ray-project/ray

johntaylor-cell · 2026-07-21T20:39:56Z

Stacked on #64911

Why

The Serve controller runs a control loop, and on every single tick it walked the full list of replicas five separate times just to rebuild lists of IDs — which replica actors are alive, which nodes have replicas on them, which replicas are running, the autoscaler's set of running replica IDs, and the rank gate's membership key. At 16K replicas those five walks dominated the loop. A CPU profile showed the loop wasn't slow because of any one expensive operation; it was slow because it kept recomputing the same five answers, and meanwhile the controller was too busy to ingest the metric reports its replicas were sending it.

The observation. Those answers only change when replica membership changes. In steady state, membership doesn't change for hours. So the walks can be computed once and cached until membership actually moves.

The mechanism. The replica container now maintains a cheap key describing its own contents, and each of those derived collections is cached against that key. When the key is unchanged, the cached answer is returned without touching a single replica.

What

ReplicaStateContainer gains a membership key: the summed per-replica hash of its
{replica id -> state} content, maintained incrementally on add/pop/remove, and
returned paired with the exact replica count. It is a hash of the content rather than a
counter of mutations because the health and migration passes pop a whole bucket and
re-add every replica to the state it came from: gang deployments take that path on
every tick, and every other deployment takes it whenever any node in the cluster is
draining, which is 5 mutations per otherwise-idle tick at 4 replicas. A counter reads
that churn as a membership change and gives back none of the win. Summing per-replica
terms makes pop-and-re-add cancel exactly, while a real arrival, departure or state
transition still changes it.
Unlike [serve] Gate rank-consistency check on replica membership changes #64911's id-set comparison, which was exact, this is a hash: two memberships
of the same size collide with probability ~2^-64. The pairing with the exact count
removes every collision where cardinality differs. The asymmetry worth knowing is that
a stale memo self-corrects on the next real membership change, whereas a skipped rank
pass would not, which is why the count is carried and why the fingerprint is pinned to
a full recompute in the tests.
The derived collections memoize on that key at both DeploymentState and
DeploymentStateManager level, through one shared _memoized_walk helper.
Caching disarms while any STARTING, UPDATING or RECOVERING replicas exist: the
startup check calls check_ready() for all three, and that rewrites
actor_id/node_id in place without a container mutation. During ramp, rollout and
recovery every getter recomputes exactly as before; the cache engages only at steady
state, which is where the waste was.
The autoscaler skips its id-set rebuild when handed the same (cached) list object it
already processed. That is safe because _running_replicas and
_cached_running_replica_strs are only ever written together and both start empty, so
the pair cannot desync — note that re-registering a deployment reuses the existing
DeploymentAutoscalingState rather than presenting a new list object.
#64911's membership key and its
error backoff both become this key — no walk at all, and the gate's check now sits
before the get() list copy instead of after it.

Two deliberate deltas from the gate as it stands in #64911: the empty-membership early
return now caches before returning, and the errored-pass path clears the key.

Cached collections are shared objects, so the getters return them frozen
(FrozenSet/Tuple) and the one caller that needs a mutable set copies at its own
boundary; every other caller only reads them.

Results

Isolated microbench of the three DeploymentState walks at 16K replicas:
3.23 ms → 0.008 ms per tick (real DeploymentReplica property chains cost
more than the benchmark stand-ins, so live savings are larger).

On the full harness (16K replicas, single deployment, pinned, steady state,
36 samples): control loop 352.6 ms → 42.9 ms, deployment-state update
253.8 ms → 17.1 ms, 1.36 → 4.99 loops/s — and the first run on this
harness to place all 16,384 replicas and complete cleanly; without this change
the controller starves its own report ingest at that scale. (Composite numbers
measured with the push-health stack on top; isolated this-PR-vs-base cells on the
same harness are running and will replace this table before review.)

Testing

The key changes only on a real membership change: empty pop/remove do not change it,
nor does popping a bucket and re-adding every replica to the state it came from, while
a transition, an arrival and a departure each do. A separate test recomputes the key
from every bucket and asserts it matches after a mixed add/pop/remove/re-bucket
sequence, so a future bucket write that forgets to maintain it fails loudly.
Getters memoize and invalidate on mutation; STARTING/UPDATING/RECOVERING disarms
caching; autoscaler identity-skip (same object skipped, new object rebuilt).
Churn coverage, since that is what a counter got wrong: the memos and [serve] Gate rank-consistency check on replica membership changes #64911's rank
gate both hold across ticks with a node draining elsewhere in the cluster, and across
steady-state ticks of a gang deployment. Each of these witnesses the churn through
bucket-list identity so none can pass vacuously, and each is verified to fail if the
key's per-replica terms are made non-cancelling.
An errored pass drops the previously validated key, so a membership it validated
before a failure is rechecked rather than skipped forever.
test_deployment_state.py + test_autoscaling_policy.py: 287 tests, green under the
default env and all four CI env variants (_with_pack_scheduling, _metr_disab,
_metr_agg_at_controller, _metr_agg_at_controller_and_replicas).

gemini-code-assist

Code Review

This pull request introduces performance optimizations to Ray Serve's deployment state management by caching and memoizing expensive per-tick id-collection walks (such as alive replica actor IDs, running replica IDs, and active node IDs) using a mutation version tracker on the replica container. It also optimizes the rank-consistency check to run only when replica membership changes. The reviewer feedback focuses on ensuring backward compatibility during rolling upgrades or controller recoveries by defensively using getattr to access newly introduced attributes on deserialized objects, and preventing state corruption by returning immutable collections (frozenset and tuple) from cached methods.

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit c7f6a48. Configure here.}

…4911) ## [serve] Gate rank-consistency check on replica membership changes ### Why The Ray Serve controller runs on the head node, continuously reconciling deployments and making autoscaling decisions. `check_rank_consistency_and_reassign_minimally` runs at the tail of every `DeploymentState.update()` — once per deployment per control-loop tick, which at measured loop rates is several times a second. The pass is O(N) in replicas with large constant factors: it materializes the active-key set and the rank-key set, takes set differences both ways, copies the entire `_ranks` dict, tallies per-rank counts across every active key, and sorts all rank values — then the manager above it regroups replicas per node. In steady state every one of those passes re-derives an identical answer. At 16K, py-spy showed it monopolizing the event loop and starving handle-report ingest for 20 s+. Ranks can only become inconsistent when replica membership changes, so gate the pass on the active replica-id set: run it when the set differs from the last checked one, skip when it does not. Transitions are unaffected — membership churn during rollouts and autoscaling changes the set, so the pass still runs exactly when it can matter. The guards that were already there (deployment not HEALTHY, or any STARTING replica) run first, before the replica list is materialized. `RAY_SERVE_FAIL_ON_RANK_ERROR` defaults off, so in production the pass can log an error and return a safe default. When that happens the membership is deliberately **not** cached and the check is retried next tick, rather than latching a membership the pass never validated. The error flag is read before the reassignment reconfigure, which calls back into the rank manager and resets it. The gate's own cost is the remaining O(N) here: building the id set measures 1.9 ms at 16,384 replicas, ~4.5% of the optimized loop. #64910, stacked on this PR, replaces the set comparison with a `ReplicaStateContainer` mutation-version integer, which removes it. ### What - Extract the pass into `_maybe_check_rank_consistency()`. - Gate it on the set of active replica ids: if the set is unchanged since the pass last ran, skip it. - Only record a membership as checked when the pass did not swallow an error. `RAY_SERVE_FAIL_ON_RANK_ERROR` is off by default, so the pass can catch an exception and return `[]`; caching that would mean a stable-but-inconsistent deployment never retries. `DeploymentRankManager` now records whether the last rank op errored. - The existing guards are preserved verbatim: only when the deployment is HEALTHY, and never while STARTING replicas exist (the node-migration case documented in the original comment). ### Why this is safe Rank consistency can only be violated by a membership change — replicas added, removed, or replaced — and every such change alters the active id set, so the pass still runs on exactly the ticks where it can find work. Rank release and removal from `_replicas` happen in the same `pop(states=[STOPPING])` branch, so the id set and the rank table cannot drift apart across ticks. The per-node and node-rank passes group off `_replica_to_node`, which is only mutated by rank assign/recover/release, so a fixed replica-id set implies a fixed node grouping. Set comparison is exact — there is no fingerprint collision to reason about. The residual cost is the O(N) set construction itself (~1.9 ms/tick at 16K replicas); the follow-up #64910 replaces it with an exact `ReplicaStateContainer` mutation counter, making the gate O(1). ### Results - Follow-up profile of the same 16K workload: the rank pass no longer appears in controller CPU samples; report ingest resumes (stale-report drops stop). - Composite (with the rest of the optimization stack): 16K steady-state control loop 352.6 ms → 42.9 ms. ### Testing - 6 new unit tests. Four over the gate itself: runs once then skips on stable membership; re-runs on membership change; a swallowed rank error is not cached and retries next tick; skips entirely (without recording a membership) while STARTING replicas exist. Two over a real `DeploymentRankManager(fail_on_rank_error=False)` — the production default — covering that a swallowed error is not cached, and that the error flag is read before `_reconfigure_replicas_with_new_ranks` can clear it. - Full `test_deployment_state.py` suite green (237 tests). <img width="1248" height="728" alt="perf_A6_loop_haproxy_on" src="https://github.com/user-attachments/assets/34758e31-6c04-46f9-958c-d6b34d3eae32" /> Signed-off-by: john.taylor <john.taylor@anyscale.com> Co-authored-by: Claude <noreply@anthropic.com>

… fingerprint The per-tick id walks (alive-actor-ids, active-node-ids, running-ids, the autoscaler id-set rebuild) recompute identical results on every steady-state tick. Memoize each on a ReplicaStateContainer membership fingerprint, and disarm while STARTING or RECOVERING replicas exist, since their actor_id/actor_node_id materialize in place without touching the container. The fingerprint is the summed per-replica hash of the container's {replica id -> state} content rather than a mutation counter, because the health and migration passes pop a whole bucket and re-add every replica to the state it came from. A counter reads that churn as a membership change, and one node draining anywhere in the cluster is enough to trigger it: at 4 replicas it bumps 5 times per otherwise-idle tick, which invalidates all three memos and re-runs the O(N) rank-consistency pass on every tick. Gang deployments health-check through that same pop-and-re-add path on every tick, so for them a counter never holds at all. Summing per-replica terms makes pop-and-re-add cancel exactly, while a real arrival, departure or state transition still changes it. The same key replaces the id-set key of the rank-consistency gate, which keeps that gate O(1) per tick without losing the content-based semantics that let it hold under churn. It changes strictly more often than the id set it replaces -- it includes state -- so the gate can never skip a pass the id set would have run. It is paired with the exact replica count, because unlike an id-set comparison it is a hash: memberships of the same size collide with probability ~2^-64, and while a stale memo self-corrects on the next real change, a skipped rank pass would not. Caching disarms while STARTING, UPDATING or RECOVERING replicas exist: the startup check calls check_ready() for all three, and that rewrites actor_id/node_id in place without a membership change. One delta from the gate as it stands in the base PR: the empty-membership early return now caches before returning. Manager-level getters return frozensets so the memo cannot be mutated by callers (with set() at the one mutating caller), and AutoscalingContext.running_replicas is copied to a list at the public boundary. The autoscaler's identity skip is safe because _running_replicas and _cached_running_replica_strs are only ever written together and both start empty, so the pair cannot desync -- note that re-registering a deployment reuses the existing DeploymentAutoscalingState rather than presenting a new list object. Signed-off-by: john.taylor <john.taylor@anyscale.com> Co-Authored-By: Claude <noreply@anthropic.com>

…y-project#64911) ## [serve] Gate rank-consistency check on replica membership changes ### Why The Ray Serve controller runs on the head node, continuously reconciling deployments and making autoscaling decisions. `check_rank_consistency_and_reassign_minimally` runs at the tail of every `DeploymentState.update()` — once per deployment per control-loop tick, which at measured loop rates is several times a second. The pass is O(N) in replicas with large constant factors: it materializes the active-key set and the rank-key set, takes set differences both ways, copies the entire `_ranks` dict, tallies per-rank counts across every active key, and sorts all rank values — then the manager above it regroups replicas per node. In steady state every one of those passes re-derives an identical answer. At 16K, py-spy showed it monopolizing the event loop and starving handle-report ingest for 20 s+. Ranks can only become inconsistent when replica membership changes, so gate the pass on the active replica-id set: run it when the set differs from the last checked one, skip when it does not. Transitions are unaffected — membership churn during rollouts and autoscaling changes the set, so the pass still runs exactly when it can matter. The guards that were already there (deployment not HEALTHY, or any STARTING replica) run first, before the replica list is materialized. `RAY_SERVE_FAIL_ON_RANK_ERROR` defaults off, so in production the pass can log an error and return a safe default. When that happens the membership is deliberately **not** cached and the check is retried next tick, rather than latching a membership the pass never validated. The error flag is read before the reassignment reconfigure, which calls back into the rank manager and resets it. The gate's own cost is the remaining O(N) here: building the id set measures 1.9 ms at 16,384 replicas, ~4.5% of the optimized loop. ray-project#64910, stacked on this PR, replaces the set comparison with a `ReplicaStateContainer` mutation-version integer, which removes it. ### What - Extract the pass into `_maybe_check_rank_consistency()`. - Gate it on the set of active replica ids: if the set is unchanged since the pass last ran, skip it. - Only record a membership as checked when the pass did not swallow an error. `RAY_SERVE_FAIL_ON_RANK_ERROR` is off by default, so the pass can catch an exception and return `[]`; caching that would mean a stable-but-inconsistent deployment never retries. `DeploymentRankManager` now records whether the last rank op errored. - The existing guards are preserved verbatim: only when the deployment is HEALTHY, and never while STARTING replicas exist (the node-migration case documented in the original comment). ### Why this is safe Rank consistency can only be violated by a membership change — replicas added, removed, or replaced — and every such change alters the active id set, so the pass still runs on exactly the ticks where it can find work. Rank release and removal from `_replicas` happen in the same `pop(states=[STOPPING])` branch, so the id set and the rank table cannot drift apart across ticks. The per-node and node-rank passes group off `_replica_to_node`, which is only mutated by rank assign/recover/release, so a fixed replica-id set implies a fixed node grouping. Set comparison is exact — there is no fingerprint collision to reason about. The residual cost is the O(N) set construction itself (~1.9 ms/tick at 16K replicas); the follow-up ray-project#64910 replaces it with an exact `ReplicaStateContainer` mutation counter, making the gate O(1). ### Results - Follow-up profile of the same 16K workload: the rank pass no longer appears in controller CPU samples; report ingest resumes (stale-report drops stop). - Composite (with the rest of the optimization stack): 16K steady-state control loop 352.6 ms → 42.9 ms. ### Testing - 6 new unit tests. Four over the gate itself: runs once then skips on stable membership; re-runs on membership change; a swallowed rank error is not cached and retries next tick; skips entirely (without recording a membership) while STARTING replicas exist. Two over a real `DeploymentRankManager(fail_on_rank_error=False)` — the production default — covering that a swallowed error is not cached, and that the error flag is read before `_reconfigure_replicas_with_new_ranks` can clear it. - Full `test_deployment_state.py` suite green (237 tests). <img width="1248" height="728" alt="perf_A6_loop_haproxy_on" src="https://github.com/user-attachments/assets/34758e31-6c04-46f9-958c-d6b34d3eae32" /> Signed-off-by: john.taylor <john.taylor@anyscale.com> Co-authored-by: Claude <noreply@anthropic.com>

…y-project#64911) ## [serve] Gate rank-consistency check on replica membership changes ### Why The Ray Serve controller runs on the head node, continuously reconciling deployments and making autoscaling decisions. `check_rank_consistency_and_reassign_minimally` runs at the tail of every `DeploymentState.update()` — once per deployment per control-loop tick, which at measured loop rates is several times a second. The pass is O(N) in replicas with large constant factors: it materializes the active-key set and the rank-key set, takes set differences both ways, copies the entire `_ranks` dict, tallies per-rank counts across every active key, and sorts all rank values — then the manager above it regroups replicas per node. In steady state every one of those passes re-derives an identical answer. At 16K, py-spy showed it monopolizing the event loop and starving handle-report ingest for 20 s+. Ranks can only become inconsistent when replica membership changes, so gate the pass on the active replica-id set: run it when the set differs from the last checked one, skip when it does not. Transitions are unaffected — membership churn during rollouts and autoscaling changes the set, so the pass still runs exactly when it can matter. The guards that were already there (deployment not HEALTHY, or any STARTING replica) run first, before the replica list is materialized. `RAY_SERVE_FAIL_ON_RANK_ERROR` defaults off, so in production the pass can log an error and return a safe default. When that happens the membership is deliberately **not** cached and the check is retried next tick, rather than latching a membership the pass never validated. The error flag is read before the reassignment reconfigure, which calls back into the rank manager and resets it. The gate's own cost is the remaining O(N) here: building the id set measures 1.9 ms at 16,384 replicas, ~4.5% of the optimized loop. ray-project#64910, stacked on this PR, replaces the set comparison with a `ReplicaStateContainer` mutation-version integer, which removes it. ### What - Extract the pass into `_maybe_check_rank_consistency()`. - Gate it on the set of active replica ids: if the set is unchanged since the pass last ran, skip it. - Only record a membership as checked when the pass did not swallow an error. `RAY_SERVE_FAIL_ON_RANK_ERROR` is off by default, so the pass can catch an exception and return `[]`; caching that would mean a stable-but-inconsistent deployment never retries. `DeploymentRankManager` now records whether the last rank op errored. - The existing guards are preserved verbatim: only when the deployment is HEALTHY, and never while STARTING replicas exist (the node-migration case documented in the original comment). ### Why this is safe Rank consistency can only be violated by a membership change — replicas added, removed, or replaced — and every such change alters the active id set, so the pass still runs on exactly the ticks where it can find work. Rank release and removal from `_replicas` happen in the same `pop(states=[STOPPING])` branch, so the id set and the rank table cannot drift apart across ticks. The per-node and node-rank passes group off `_replica_to_node`, which is only mutated by rank assign/recover/release, so a fixed replica-id set implies a fixed node grouping. Set comparison is exact — there is no fingerprint collision to reason about. The residual cost is the O(N) set construction itself (~1.9 ms/tick at 16K replicas); the follow-up ray-project#64910 replaces it with an exact `ReplicaStateContainer` mutation counter, making the gate O(1). ### Results - Follow-up profile of the same 16K workload: the rank pass no longer appears in controller CPU samples; report ingest resumes (stale-report drops stop). - Composite (with the rest of the optimization stack): 16K steady-state control loop 352.6 ms → 42.9 ms. ### Testing - 6 new unit tests. Four over the gate itself: runs once then skips on stable membership; re-runs on membership change; a swallowed rank error is not cached and retries next tick; skips entirely (without recording a membership) while STARTING replicas exist. Two over a real `DeploymentRankManager(fail_on_rank_error=False)` — the production default — covering that a swallowed error is not cached, and that the error flag is read before `_reconfigure_replicas_with_new_ranks` can clear it. - Full `test_deployment_state.py` suite green (237 tests). <img width="1248" height="728" alt="perf_A6_loop_haproxy_on" src="https://github.com/user-attachments/assets/34758e31-6c04-46f9-958c-d6b34d3eae32" /> Signed-off-by: john.taylor <john.taylor@anyscale.com> Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: 400Ping <jiekaichang@apache.org>

github-actions · 2026-08-25T00:26:09Z

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

johntaylor-cell requested a review from a team as a code owner July 21, 2026 20:39

gemini-code-assist Bot reviewed Jul 21, 2026

View reviewed changes

johntaylor-cell force-pushed the serve-a7-idwalk-cache branch from 2c9f04a to b5ff9be Compare July 21, 2026 23:01

ray-gardener Bot added the serve Ray Serve Related Issue label Jul 22, 2026

johntaylor-cell self-assigned this Jul 22, 2026

johntaylor-cell added the go add ONLY when ready to merge, run all tests label Jul 22, 2026

johntaylor-cell force-pushed the serve-a7-idwalk-cache branch 3 times, most recently from 0073e32 to a110642 Compare July 22, 2026 17:46

johntaylor-cell added the performance label Jul 22, 2026

johntaylor-cell mentioned this pull request Jul 26, 2026

[serve] Gate rank-consistency check on replica membership changes #64911

Merged

johntaylor-cell force-pushed the serve-a7-idwalk-cache branch 2 times, most recently from d169329 to abe8f6a Compare July 27, 2026 17:56

cursor Bot reviewed Jul 27, 2026

View reviewed changes

Comment thread python/ray/serve/_private/deployment_state.py Outdated

johntaylor-cell force-pushed the serve-a7-idwalk-cache branch 2 times, most recently from a846bb0 to 78aa66d Compare July 27, 2026 19:12

cursor Bot reviewed Jul 27, 2026

View reviewed changes

Comment thread python/ray/serve/tests/unit/test_deployment_state.py Outdated

johntaylor-cell force-pushed the serve-a7-idwalk-cache branch from 78aa66d to c7f6a48 Compare July 27, 2026 19:50

cursor Bot reviewed Jul 27, 2026

View reviewed changes

Comment thread python/ray/serve/_private/deployment_state.py Outdated

johntaylor-cell force-pushed the serve-a7-idwalk-cache branch 4 times, most recently from e9917d9 to 0465c10 Compare July 27, 2026 21:11

eicherseiji mentioned this pull request Jul 27, 2026

[core][telemetry] Reduce cardinality of Serve metrics to node level #65053

Closed

johntaylor-cell force-pushed the serve-a7-idwalk-cache branch 4 times, most recently from 2a34b66 to 7a81433 Compare July 28, 2026 02:31

johntaylor-cell requested a review from abrarsheikh July 28, 2026 19:07

johntaylor-cell force-pushed the serve-a7-idwalk-cache branch 7 times, most recently from 6ddcee6 to 25182fc Compare July 30, 2026 16:14

johntaylor-cell force-pushed the serve-a7-idwalk-cache branch from 25182fc to ffb957c Compare August 10, 2026 16:11

github-actions Bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Aug 25, 2026

johntaylor-cell added unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it. and removed stale The issue is stale. It will be closed within 7 days unless there are further conversation labels Aug 25, 2026

Navigation Menu

Uh oh!

[serve] Cache per-tick replica id walks behind a container mutation version - #64910

[serve] Cache per-tick replica id walks behind a container mutation version#64910
johntaylor-cell wants to merge 1 commit into
ray-project:masterfrom
johntaylor-cell:serve-a7-idwalk-cache

johntaylor-cell commented Jul 21, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

johntaylor-cell commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Results

Testing

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

johntaylor-cell commented Jul 21, 2026 •

edited

Loading