Shard Auto-Split Design

[merlimat]

[mattisonchao]

Hi @merlimat

I've read the proposal and left some comments:

Configuration Parts.

shardManagement:
maxShardsPerNamespace: 64 # safety guard-rail (default: 64)
autoSplit:
enabled: false # default: false
maxShardSizeMB: 1024 # 1 GiB — split when DB disk usage exceeds this
maxThroughputOps: 10000 # 10K ops/sec — combined reads + writes threshold
stabilizationPeriod: 1m # how long a threshold must be exceeded before triggering
cooldownPeriod: 5m # minimum time between consecutive splits

We could consider the hierarchy name, and then the configuration can also be used at the namespace level in the future.

for example:

> 
shardManagement:
  maxShardsNum: 64         
  autoSplit:
    enabled: false                           # default: false
    maxShardSize: "1GiB"             # 1 GiB — 
    maxThroughputOps: 10000    # 10K ops/sec — combined reads + writes threshold
    stabilizationPeriod: "1m"            # how long a threshold must be exceeded before triggering
    cooldownPeriod: "5m"               # minimum time between consecutive splits

Some points:

1. We could reuse the whole struct for namespace-level in the future.
2. Use maxShardSize with the unit format, like k8s resource.

Auto Split Trigger

1. We should consider changing the load balancer logic to skip the splitting/merging of shards to avoid interference (it seems like event loop shard controller has done this? ).
2. We should consider making the duty clear for both components.
   - LoadBalancer: Focusing on the shards balancing, moving and leader balancing by placement policy and traffic.
   - AutoSplitMonitor: Focusing on the traffic/disk-based monitoring. which will split the high traffic/usage shards.
   Therefore, those two components should ultimately make the entire cluster load-balanced.

Observability

• consider adding the duration of splitting.