WIP: Minimal refactoring to have a single shard by shmuelk · Pull Request #2658 · kubernetes-sigs/gateway-api-inference-extension

shmuelk · 2026-03-22T10:47:36Z

What type of PR is this?
/kind cleanup

What this PR does / why we need it:
The Flow Control component is a critical component in the Endpoint Picker (EPP), enabling it to throttle workloads thus preventing over committing Model Server resources.

Issue #2628 was created to describe a set of simplifications to the Flow Control layer. This PR is the first in a series to implement issue #2628.

In particular this PR changes the Flow Control layer to only have a single shard.

Which issue(s) this PR fixes:
Refs #2628

Does this PR introduce a user-facing change?:

NONE

netlify · 2026-03-22T10:47:44Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`cf2de59`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/69ca82e40e285b000708c8f7
😎 Deploy Preview	https://deploy-preview-2658--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Gregory-Pereira · 2026-03-22T17:16:10Z

cc @RishabhSaini

nirrozenbaum · 2026-03-22T20:26:09Z

/cc @LukeAVanDrie

LukeAVanDrie

Thanks @shmuelk! This LGTM. I noticed one place we can simplify the new createShard method, else I only have a few nits.

LukeAVanDrie · 2026-03-23T16:28:04Z

pkg/epp/flowcontrol/registry/registry.go

-	allShards      []*registryShard // Cached, sorted combination of Active and Draining shards
-	nextShardID    uint64
+	mu    sync.RWMutex
+	shard *registryShard


nit: This is currently read in ActiveShards() and ShardStats() without acquiring fr.mu.RLock(). With dynamic sharding removed, fr.shard is initialized once and never mutated, making these lock-free reads completely safe from data races.

Could we move the shard *registryShard field of the "Administrative state (protected by mu)" block and up to the "Immutable dependencies" block?

I will do this in the next PR

LukeAVanDrie · 2026-03-23T16:30:08Z

pkg/epp/flowcontrol/registry/registry.go

-	}
-
+// createShard creates the shard.
+func (fr *FlowRegistry) createShard() error {


The entire block of code in here that iterates over fr.flowStates.Range(...) to build allComponents and synchronizeFlow is actually dead code.

Because createShard() is exclusively called during NewFlowRegistry() before the EPP accepts any connections, fr.flowStates is guaranteed to be empty.

You can simplify this initialization method to just:

func (fr *FlowRegistry) createShard() error { fr.mu.Lock() defer fr.mu.Unlock() partitionedConfig := fr.config.partition(0, 1) fr.shard = newShard("shard-0", partitionedConfig, fr.logger, fr.propagateStatsDelta) return nil }

LukeAVanDrie · 2026-03-23T16:37:36Z

pkg/epp/flowcontrol/registry/registry.go

-
 // repartitionShardConfigsLocked updates the configuration for all active shards.
 // Expects the registry's write lock to be held.
 func (fr *FlowRegistry) repartitionShardConfigsLocked() {


Note for other reviewers... This looks weird considering a single-shard view, but we must preserve it for now. When ensurePriorityBand dynamically creates a new band, this partition path acts as a deep-copy mechanism to push the mutated registry config down to the isolated shard state.

In a follow-up PR (if/when we eliminate the boundary between registryShard and FlowRegistry entirely), we can have everything reference a single unified Config, allowing us to drop this path entirely.

LukeAVanDrie · 2026-03-23T16:39:52Z

pkg/epp/flowcontrol/registry/registry.go

 	defer fr.mu.RUnlock()

-	components, err := fr.buildFlowComponents(key, len(fr.allShards))
+	components, err := fr.buildFlowComponents(key, 1)


nit: Consider updating buildFlowComponents to drop the numInstances arg and just return a single (flowComponents, error) tuple. Fine if we want to defer this to a different PR though.

I will do this in the next PR

LukeAVanDrie · 2026-03-23T16:44:45Z

/lgtm

kfswain · 2026-03-23T16:59:18Z

pkg/epp/flowcontrol/registry/connection.go

-	for i, s := range c.registry.activeShards {
-		shardsCopy[i] = s
-	}
+	shardsCopy := make([]contracts.RegistryShard, 1)


Are we going to remove the concept of a shard in a later PR?

Yes, there will be no "data-parallel" concept anymore. See the attached issue for more context.

For these PRs though, as long as the FC layer has 0 regressions between revisions, I want to get these in even if there are some minor stylistic/semantic improvements that could be made. This code should look significantly different after these are all in, so I think it is most expedient to focus on polish at the end of the refactoring effort rather than at each intermediary step.

That's totally fine with me, just making sure we are all pointed in the same direction

kfswain · 2026-03-23T17:11:20Z

/approve

k8s-ci-robot · 2026-03-23T17:11:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kfswain, LukeAVanDrie, shmuelk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [kfswain]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

shmuelk · 2026-03-24T12:36:22Z

@LukeAVanDrie and @kfswain thank you for the reviews.

I would remove the WIP (i.e. hold) except that in some attempts to compare performance, I see some strange results. I'm trying to get to the bottom of it.

ahg-g · 2026-03-30T13:33:05Z

rebase pls?

ahg-g · 2026-03-30T13:33:34Z

I would remove the WIP (i.e. hold) except that in some attempts to compare performance, I see some strange results. I'm trying to get to the bottom of it.

Still seeing a regression in perf?

shmuelk · 2026-03-30T13:54:07Z

I am running a slightly modified version of Like's benchmark (only one shard only up to 5000 Items in the queue)

The original script I saw in Luke's PR ran the tests each for one sec and repeated five times.

This run showed some regressions in cases where there were 5000 workers. 5000 goroutines didin't start well in one second.

I have since run the tests each for five seconds and again repeated five times. The results are much better, but there are spikes.

Here is an Excel spreadsheet of my results:
baseline-base-step1-compare.xlsx

The baseline numbers are an average of four runs of the benchmark script. The step one (the code from this PR) numbers are from eight runs of the benchmark script. The ns/op and d/s numbers are averaged over the tests in question. I have added median and max values for the tests in the step one numbers.

Signed-off-by: Shmuel Kallner <kallner@il.ibm.com>

k8s-ci-robot · 2026-03-30T14:04:20Z

New changes are detected. LGTM label has been removed.

LukeAVanDrie · 2026-03-30T15:47:05Z

This run showed some regressions in cases where there were 5000 workers. 5000 goroutines didin't start well in one second.

I have since run the tests each for five seconds and again repeated five times. The results are much better, but there are spikes.

The test amortizes the first-time flow / priority band instantiation across the bench time, so running it for longer durations (5s) vs (1s) is actually preferable to smooth that out and look at steady-state performance under load.

shmuelk · 2026-03-31T08:52:13Z

I have run my performance benchmarks on my laptop which is:

Chip: Apple M4 Max
Total Number of Cores: 16 (12 performance and 4 efficiency)
Memory: 64 GB

The tests above used all sixteen cores, which apparently are not all equal.

I have run an additional set of tests using only twelve cores, in an attempt to not use the "efficiency cores".

Here are the results:
baseline-base-step1-compare.xlsx

k8s-ci-robot · 2026-04-01T19:58:36Z

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. labels Mar 22, 2026

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 22, 2026

k8s-ci-robot requested review from danehans and liu-cong March 22, 2026 10:47

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Mar 22, 2026

shmuelk changed the title ~~WIP: Minimal refacctoring to have a single shard~~ WIP: Minimal refactoring to have a single shard Mar 22, 2026

k8s-ci-robot requested a review from LukeAVanDrie March 22, 2026 20:26

LukeAVanDrie approved these changes Mar 23, 2026

View reviewed changes

k8s-ci-robot assigned LukeAVanDrie Mar 23, 2026

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 23, 2026

kfswain reviewed Mar 23, 2026

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 23, 2026

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 29, 2026

Minimal refacctoring to have a single shard

cf2de59

Signed-off-by: Shmuel Kallner <kallner@il.ibm.com>

shmuelk force-pushed the fc-refactor-1 branch from 73ea409 to cf2de59 Compare March 30, 2026 14:04

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 30, 2026

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 30, 2026

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 1, 2026

Conversation

shmuelk commented Mar 22, 2026

Uh oh!

netlify bot commented Mar 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

Gregory-Pereira commented Mar 22, 2026

Uh oh!

nirrozenbaum commented Mar 22, 2026

Uh oh!

LukeAVanDrie left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LukeAVanDrie commented Mar 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfswain commented Mar 23, 2026

Uh oh!

k8s-ci-robot commented Mar 23, 2026

Uh oh!

shmuelk commented Mar 24, 2026

Uh oh!

ahg-g commented Mar 30, 2026

Uh oh!

ahg-g commented Mar 30, 2026

Uh oh!

shmuelk commented Mar 30, 2026

Uh oh!

k8s-ci-robot commented Mar 30, 2026

Uh oh!

LukeAVanDrie commented Mar 30, 2026

Uh oh!

shmuelk commented Mar 31, 2026

Uh oh!

k8s-ci-robot commented Apr 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

netlify bot commented Mar 22, 2026 •

edited

Loading