Long-context support: configurable context length (currently hard-capped at 32768) and paged KV cache
1. Context length is baked at 32768 with no way to raise it
The chat models default to a 32768 context, and there is no user-facing way to increase it. It is hard-coded in the model's genai_config.json in the FL cache:
~/.foundry/cache/models/Microsoft/qwen2.5-coder-7b-instruct-generic-cpu-4/v4/genai_config.json
model.context_length = 32768
search.max_length = 32768
foundry model info -o json reports the same contextLength: 32768, and neither foundry model load nor foundry config exposes any option to change it. So a model like Qwen2.5-Coder-7B — which natively supports up to 131072 (128k) via YaRN — is usable only up to 32768 through Foundry Local, because the shipped ONNX export and its genai_config.json cap it there.
Requests:
- Ship (or let users select) longer-context variants of catalog models where the base model supports it (e.g. 128k Qwen2.5), and/or
- Expose a context-length option (CLI flag / config / per-request) that raises
context_length / search.max_length for models whose weights support extended context (YaRN/rope-scaling), instead of requiring a re-export.
2. No paged KV cache — KV memory is pre-allocated for the full context
ORT-GenAI is configured with past_present_share_buffer: true, i.e. a single contiguous KV buffer pre-allocated to the maximum context length. There is no paged-attention / paged-KV-cache option. Consequences:
- KV memory is reserved up front for the full context even for short conversations.
- Large context lengths scale KV memory linearly with the cap, which is a real barrier to the 128k+ contexts requested above — a pre-allocated 128k KV buffer per sequence is very memory-heavy.
Request: support a paged KV cache / paged-attention mode (vLLM-style block paging), so long-context and concurrent sequences are memory-efficient rather than requiring a worst-case contiguous allocation.
Why this matters
Agentic and RAG workloads exhaust 32768 quickly (tool schemas + tool outputs + history). Raising the usable context and paging the KV cache are the two blockers to using Foundry Local for longer-context agent workflows.
Environment
- Foundry Local
0.10.0; ORT-GenAI backend; models from the FL catalog.
Long-context support: configurable context length (currently hard-capped at 32768) and paged KV cache
1. Context length is baked at 32768 with no way to raise it
The chat models default to a 32768 context, and there is no user-facing way to increase it. It is hard-coded in the model's
genai_config.jsonin the FL cache:foundry model info -o jsonreports the samecontextLength: 32768, and neitherfoundry model loadnorfoundry configexposes any option to change it. So a model like Qwen2.5-Coder-7B — which natively supports up to 131072 (128k) via YaRN — is usable only up to 32768 through Foundry Local, because the shipped ONNX export and itsgenai_config.jsoncap it there.Requests:
context_length/search.max_lengthfor models whose weights support extended context (YaRN/rope-scaling), instead of requiring a re-export.2. No paged KV cache — KV memory is pre-allocated for the full context
ORT-GenAI is configured with
past_present_share_buffer: true, i.e. a single contiguous KV buffer pre-allocated to the maximum context length. There is no paged-attention / paged-KV-cache option. Consequences:Request: support a paged KV cache / paged-attention mode (vLLM-style block paging), so long-context and concurrent sequences are memory-efficient rather than requiring a worst-case contiguous allocation.
Why this matters
Agentic and RAG workloads exhaust 32768 quickly (tool schemas + tool outputs + history). Raising the usable context and paging the KV cache are the two blockers to using Foundry Local for longer-context agent workflows.
Environment
0.10.0; ORT-GenAI backend; models from the FL catalog.