Add KV-cache file prefetch plugin#1156
Conversation
|
🚨 Unsigned commits detected! Please sign your commits. For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation. |
github-actions
Bot
added
the
size/XXL
leipanhz
force-pushed
the
feat/kv-cache-prefetch-plugin
branch
5 times, most recently
from
4633c91 to
9a7d956
Compare
elevran
added
the
hold
Introduce a PreRequest plugin that proactively prefetches a request's KV-cache files from a shared file system (e.g. IBM Storage Scale) before the request reaches the GPU pod, promoting cold blocks to a closer tier to reduce inference latency. The plugin obtains the request's full-width block-hash digests from the precise-prefix-cache-producer via GetEngineKeysAndDigestsForRequest and maps each digest to its on-disk llm-d-fs-connector path, then reads the head of each file on a worker pool to trigger the storage tier promotion. File paths are constructed entirely from operator-supplied config rather than filesystem discovery: rootDir, modelName, the fs-connector digest, groupIdx, gpuBlocksPerFile, and the parallel sizes (tp/pp/pcp/dcp). The rank space is iterated over [0, tp*pp*pcp*dcp); a file that is not yet present is skipped without error. Engine-key/digest computation lives on the producer: the shared extra-features construction (multimodal hashes + cache salt) is factored into buildBlockExtraFeatures so the engine-key and digest paths taint the hash identically. Signed-off-by: Lei Pan <leipan@ibm.com> Co-authored-by: Yue Zhu <Yue.Zhu@ibm.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
leipanhz
force-pushed
the
feat/kv-cache-prefetch-plugin
branch
from
9a7d956 to
baf30cf
Compare
| // across prompts and aligned index-for-index. | ||
| func (p *Producer) GetEngineKeysAndDigestsForRequest(_ context.Context, | ||
| request *scheduling.InferenceRequest) ([]uint64, [][]byte, error) { | ||
| engineKeys, digests, err := computeBlockKeysWithDigests(p.tokenProcessor, request, p.blockSizeTokens) |
There was a problem hiding this comment.
Enabling prefetch forces hashAlgorithm: sha256-cbor on the shared producer tokenProcessor, which also switches the requestKey/scoring path to sha256.
That collapses the requestKey<-->engineKey separation and puts sha256 on the scoring/event hot path even for non-prefetch traffic. Since the router only needs vLLM-matching hashes for prefetch filenames (the index already bridges engineKey-->requestKey for lookup),
could the engine-key/digest computation use a dedicated sha256-cbor hasher, leaving the scoring requestKey hasher independent (e.g. FNV)?
There was a problem hiding this comment.
Hi @bongwoobak, you are correct that "the router only needs vLLM-matching hashes for prefetch filenames". The intention of using sha256-cbor as hashingAlgorithm is to offer user options to choose a different hashing algorithm other than the default FNV and reduce the complexity of using two hashing algorithms in the router. We had similar discussions with @vMaroon earlier, adding him here for additional comments or corrections if needed. Thank you
| type KVFilePathBaseParams struct { | ||
| RootDir string `json:"rootDir"` | ||
| ModelName string `json:"modelName"` | ||
| Digest string `json:"digest"` |
There was a problem hiding this comment.
The digest is the fingerprint of the running vLLM's cache config, pinned statically in config here.
Isn't it effectively per-deployment it shifts whenever the backend's model/dtype/parallelism or vLLM version changes (e.g. a pod rolled out with a new config/version)? A stale value then silently no-ops with no error.
Could we read it dynamically when an endpoint is (re)discovered from a vLLM metric/label or similar?
There was a problem hiding this comment.
The digest is the fingerprint of the running vLLM's cache config, pinned statically in config here.
Isn't it effectively per-deployment it shifts whenever the backend's model/dtype/parallelism or vLLM version changes (e.g. a pod rolled out with a new config/version)? A stale value then silently no-ops with no error.
Could we read it dynamically when an endpoint is (re)discovered from a vLLM metric/label or similar?
You suggestion is what we are looking for in the long run, having API support from vllm to provide config information. Currently there is no API (or metrics) available to retrieve the configuration content, a separately efffor is working to address this: vllm-project/vllm#38147. We are actively tracking this and
following up.
Detect residency from st_blocks*512 vs the logical size: skip the fault-in read when a file <= the buffer is fully resident, or when a larger file already has more than a buffer's worth allocated. Tested with IBM Storage Scale AFM, where reading the first buffer triggers a full object fetch. Signed-off-by: Lei Pan <leipan@ibm.com>
Extract submitForPrefetch and make submission unconditionally non-blocking best-effort: drop the file immediately when the queue is full. Remove the queueTimeout knob entirely -- there is no configuration that puts waiting on the request path. A dropped prefetch is a benign warm-up miss, so skipped is the queue-saturation signal. Log the per-request digest-hashing duration at DEBUG. Refresh README (non-blocking behavior, on-disk path layout, deployment config example with volume mounts, on-path latency overhead) and add TestSubmitForPrefetch. Signed-off-by: Lei Pan <leipan@ibm.com>
leipanhz
force-pushed
the
feat/kv-cache-prefetch-plugin
branch
from
739c531 to
4f44b6f
Compare
4 participants
Add KV-cache file prefetch plugin for inference requests (experimental feature)
Part 1: changes in KV-Cache
Part 2: changes in llm-d-router (current PR)
PR Description:
Introduces a new experimental feature that aims to proactively prefetch KV-cache blocks across different storage tiers before inference requests are processed by the GPU pod. The plugin extends the precise prefix cache scorer with engine key calculation to determine the storage location (file names) of KV-cache blocks that will be needed and arrange for them to be promoted to a closer storage tier to improve inference latency. The current implementation is intended for a shared file system that includes transparent access to a remote storage tier, such as IBM Storage Scale configured to off-load cold data to remote object storage. The prefetch plugin uses a concurrent worker thread pool architecture to efficiently prefetch multiple (configurable) files in parallel from remote storage to the shared file system. In a future version of the plugin this could be extended, for example, to prefetch KV-cache blocks from the file system to CPU memory on the worker node that the request is being routed to.
For this to work correctly, the plugin must be configured to use a hash algorithm for generating engine keys that matches the algorithm used by vLLM when offloading KV-cache blocks to storage. For this purpose, this work adds a configurable hashing algorithm SHA256-CBOR to the token processor as an alternative for vLLM compatibility. The SHA256-CBOR implementation supports extra keys (multimodal features) in block hash computation. In addition, this feature relies on logic derived from the llm-d-fs-connector to generate KV file names, so it currently only works with the llm-d-fs-connector.
Changes include:
New Prefetch Plugin (prefetch_prerequest_experimental.go):
Implements PreRequest interface for pre-inference file prefetching
Converts engine keys to filesystem paths using llm-d-fs-connector format
Manages worker thread pool for concurrent file prefetching (configurable workers)
Each worker reads configurable number of blocks (BlockSize x BlockCount bytes) from KV-cache files to trigger prefetch of the rest of the file from remote storage.
Supports configurable prefetch parameters (block size, concurrency, queue size)
Precise Prefix Cache Scorer Enhancement (precise_prefix_cache.go):
Add GetEngineKeysForRequest() method to extract engine keys from requests
Support multimodal features in engine key computation
Add SHA256-CBOR hashing algorithm for token processor with extra keys support
Add configuration to choose hashing function via the field name “hashAlgorithm”: FNV64a default, SHA256-CBOR for vLLM
Implement SHA256-CBOR hashing matching vLLM engine-key computation
Extend BlockExtraFeatures for multimodal content support
What type of PR is this?
/kind feature
What this PR does / why we need it:
See description in #866
Which issue(s) this PR fixes:
#866
Fixes #
Release note (write
NONEif no user-facing change):