Skip to content

Commit d68b9ff

Browse files
prekshivyasprekshivyas
authored
fix(inference): bound Spark llama.cpp request bodies (#9669)
Part of #9592. Production enforcement depends on #9600. ## Summary - cap the Muse Glimmer Spark llama.cpp recipe at 16 KiB and the Nemotron Spark recipe at 32 KiB - add an exact-size request-body qualification probe: the declared boundary must succeed, 50,000 bytes must return a structured 413, and immediate health plus a normal completion must still succeed - update the owned llama.cpp image to revision `8e7f22b67ef4667b4ddd50230771287f328cfb3f`, already declared by the Muse recipe, and permit its `--jinja` / `--chat-template-kwargs` launch arguments through the request guard - refresh the Ubuntu curl package pins from `.11` to the currently published `.12` build so the owned image remains reproducible ## Root cause and impact On DGX Spark, llama.cpp can terminate while handling an authenticated request far below the configured context window. The raw Muse runtime closed an exact 50,000-byte request, incremented Docker RestartCount, and reloaded the model. This was not an OOM. Hardware testing also disproved the initial 32 KiB candidate for Muse: an exact 32,768-byte request made the child upstream unavailable and left immediate health and continuation at 502. A conservative 16 KiB Muse boundary passed. Nemotron remains safe at 32 KiB. ## DGX Spark evidence Host: NVIDIA GB10, Ubuntu 24.04.4 LTS arm64, driver 580.159.03, Docker 29.2.1. Locally built owned image: `sha256:d1cce88e51c33a3de634caaef421cf4268a7e00ca0580260dad809cb22c13a48` | Model | Accepted request | Oversized request | RestartCount | Immediate health | Continuation | | --- | ---: | ---: | ---: | ---: | ---: | | Muse Glimmer | 16,384 bytes → 200 | 50,000 bytes → 413 `request_body_too_large` | 0 → 0 | 200 in 6 ms | 200 | | Nemotron 3 Nano | 32,768 bytes → 200 | 50,000 bytes → 413 `request_body_too_large` | 0 → 0 | 200 in 8 ms | 200 | The original raw Muse reproduction used the recipe-pinned image and exact 50,000-byte body: curl exit 52 / HTTP 000, RestartCount 0 → 1, OOMKilled false, and health returned after 5,745 ms. ## Validation - `npm run checks:repository` - `npm run typecheck:cli` - `npm --prefix nemoclaw run typecheck` - `npm run format:check` - 150 focused integration tests - 29 host-local runtime tests - complete request-guard Go test suite - pre-commit catalog, repository, hadolint, gitleaks, source-shape, and growth checks - native ARM64 CUDA image build and live Muse/Nemotron hardware qualification ## Scope boundary This change establishes model-specific safe request boundaries and qualifies the owned guard path. It does not resolve #9600: the production managed launch path still bypasses the request guard, so #9600 remains the production-enforcement dependency. Signed-off-by: Prekshi Vyas <34834085+prekshivyas@users.noreply.github.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added validation for request body size limits, including accepted and oversized request scenarios. * Oversized requests now return a clear HTTP 413 error while subsequent health and completion requests continue normally. * Added support for additional llama.cpp configuration options, including chat template parameters and Jinja templating. * **Bug Fixes** * Reduced configured request body limits to improve protection against oversized payloads. * Updated llama.cpp and runtime components for improved reliability and compatibility. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
1 parent 6fe1ed0 commit d68b9ff

10 files changed

Lines changed: 270 additions & 29 deletions

‎managed-inference/images/llama-cpp/image.yaml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ spec:
7272
- context-window
7373
- authentication
7474
- malformed-request
75+
- request-body-limit
7576
- cancellation
7677
- client-timeout
7778
- log-redaction
@@ -89,8 +90,8 @@ spec:
8990

9091
source:
9192
repository: https://github.com/ggml-org/llama.cpp
92-
revision: 22dc605c4ead20e36f447cc67b55ef87e523bd55
93-
archiveSha256: sha256:975f70723e053785e894f4e1d9cf770f2f1a7bc762fd3af174ff5635014108b6
93+
revision: 8e7f22b67ef4667b4ddd50230771287f328cfb3f
94+
archiveSha256: sha256:45a24299e7a24410624489d19924d492bc71a120fa17d9b7cb32f6d5c4f1aed0
9495

9596
cuda:
9697
developmentBase: docker.io/nvidia/cuda@sha256:ef2203909e80b8b976cfc672f7e2ae2b00bc0e25c404ee86d89e10a3802f1c52

‎managed-inference/images/llama-cpp/request-guard/main.go‎

Lines changed: 23 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -97,27 +97,29 @@ func requireExactCommandMarker(command []string, option string) error {
9797

9898
func validateSupportedCommandOptions(command []string) error {
9999
allowed := map[string]bool{
100-
"--alias": true,
101-
"--api-key-file": true,
102-
"--batch-size": true,
103-
"--cache-type-k": true,
104-
"--cache-type-v": true,
105-
"--ctx-size": true,
106-
"--flash-attn": true,
107-
"--gpu-layers": true,
108-
"--host": true,
109-
"--metrics": false,
110-
"--model": true,
111-
"--no-agent": false,
112-
"--no-mmproj": false,
113-
"--no-slots": false,
114-
"--no-ui": false,
115-
"--n-predict": true,
116-
"--parallel": true,
117-
"--port": true,
118-
"--sleep-idle-seconds": true,
119-
"--timeout": true,
120-
"--ubatch-size": true,
100+
"--alias": true,
101+
"--api-key-file": true,
102+
"--batch-size": true,
103+
"--cache-type-k": true,
104+
"--cache-type-v": true,
105+
"--chat-template-kwargs": true,
106+
"--ctx-size": true,
107+
"--flash-attn": true,
108+
"--gpu-layers": true,
109+
"--host": true,
110+
"--jinja": false,
111+
"--metrics": false,
112+
"--model": true,
113+
"--no-agent": false,
114+
"--no-mmproj": false,
115+
"--no-slots": false,
116+
"--no-ui": false,
117+
"--n-predict": true,
118+
"--parallel": true,
119+
"--port": true,
120+
"--sleep-idle-seconds": true,
121+
"--timeout": true,
122+
"--ubatch-size": true,
121123
}
122124
seen := make(map[string]bool, len(allowed))
123125
for index := 0; index < len(command); index++ {

‎managed-inference/images/llama-cpp/request-guard/main_test.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,8 @@ func TestParseConfigRequiresEveryDeclaredValue(t *testing.T) {
119119
"--port", "8082",
120120
"--api-key-file", llamaServerAPIKeyPath,
121121
"--n-predict", "4096",
122+
"--jinja",
123+
"--chat-template-kwargs", `{"reasoning_strength":"low"}`,
122124
"--no-ui",
123125
"--no-slots",
124126
"--no-mmproj",

‎managed-inference/recipes/llama-cpp.muse-glimmer-30b.spark-single.v1.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ spec:
8888
value: f16
8989
speculativeDecoding: disabled
9090
limits:
91-
maxRequestBodyBytes: 1048576
91+
maxRequestBodyBytes: 16384
9292
maxRequestHeaderBytes: 32768
9393
maxOutputTokens: 4096
9494
requestTimeoutSeconds: 900

‎managed-inference/recipes/llama-cpp.nemotron-3-nano-30b-a3b.spark-single.v1.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ spec:
1616
technology: llama.cpp
1717
source:
1818
repository: ggml-org/llama.cpp
19-
revision: 22dc605c4ead20e36f447cc67b55ef87e523bd55
19+
revision: 8e7f22b67ef4667b4ddd50230771287f328cfb3f
2020

2121
model:
2222
id: unsloth/Nemotron-3-Nano-30B-A3B-GGUF
@@ -86,7 +86,7 @@ spec:
8686
value: f16
8787
speculativeDecoding: disabled
8888
limits:
89-
maxRequestBodyBytes: 1048576
89+
maxRequestBodyBytes: 32768
9090
maxRequestHeaderBytes: 32768
9191
maxOutputTokens: 4096
9292
requestTimeoutSeconds: 900

‎scripts/checks/llama-cpp-dgx-spark-protocol-qualification.mts‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33

44
import {
55
LLAMA_CPP_DGX_SPARK_PROTOCOL_PROBES,
6+
LLAMA_CPP_DGX_SPARK_REJECTED_REQUEST_BODY_BYTES,
67
LLAMA_CPP_DGX_SPARK_REQUIRED_METRIC_SERIES,
78
type LlamaCppDgxSparkExecutionPlan,
89
type LlamaCppDgxSparkQualificationReceipt,
@@ -164,6 +165,51 @@ function jsonRequest(
164165
};
165166
}
166167

168+
function exactSizeChatRequest(model: string, targetBytes: number): string {
169+
const request = {
170+
max_tokens: 1,
171+
messages: [{ content: "", role: "user" }],
172+
model,
173+
temperature: 0,
174+
};
175+
const emptyBody = JSON.stringify(request);
176+
const contentBytes = targetBytes - new TextEncoder().encode(emptyBody).byteLength;
177+
if (contentBytes < 0) throw new Error("request-body probe target is too small");
178+
request.messages[0].content = "x".repeat(contentBytes);
179+
const body = JSON.stringify(request);
180+
if (new TextEncoder().encode(body).byteLength !== targetBytes) {
181+
throw new Error("request-body probe did not construct the exact declared size");
182+
}
183+
return body;
184+
}
185+
186+
function exactSizeJsonRequest(
187+
authorization: string,
188+
body: string,
189+
timeoutMilliseconds: number,
190+
): RequestInit {
191+
return {
192+
body,
193+
headers: {
194+
Authorization: authorization,
195+
"Content-Type": "application/json",
196+
},
197+
method: "POST",
198+
signal: requestSignal(timeoutMilliseconds),
199+
};
200+
}
201+
202+
function validateRequestBodyLimitError(value: unknown): void {
203+
if (
204+
!isRecord(value) ||
205+
!isRecord(value.error) ||
206+
value.error.code !== "request_body_too_large" ||
207+
value.error.type !== "invalid_request_error"
208+
) {
209+
throw new Error("request-body limit probe did not return the declared error contract");
210+
}
211+
}
212+
167213
function usageFrom(value: unknown): ProtocolEvidence["usage"] {
168214
if (!isRecord(value)) throw new Error("chat usage was not returned");
169215
const promptTokens = value.prompt_tokens;
@@ -727,6 +773,73 @@ export async function runLlamaCppDgxSparkProtocolQualification(options: {
727773
await expectStatus(malformedResponse, 400, bounds.maxResponseBytes, "malformed-request probe");
728774
executedProbes.add("malformed-request");
729775

776+
const acceptedRequestBytes = plan.recipe.serve.limits.maxRequestBodyBytes;
777+
const acceptedResponse = await fetchImpl(
778+
chatUrl,
779+
exactSizeJsonRequest(
780+
authorization,
781+
exactSizeChatRequest(model, acceptedRequestBytes),
782+
timeoutMilliseconds,
783+
),
784+
);
785+
await expectStatus(
786+
acceptedResponse,
787+
200,
788+
bounds.maxResponseBytes,
789+
"request-body boundary probe",
790+
);
791+
792+
const rejectedResponse = await fetchImpl(
793+
chatUrl,
794+
exactSizeJsonRequest(
795+
authorization,
796+
exactSizeChatRequest(model, LLAMA_CPP_DGX_SPARK_REJECTED_REQUEST_BODY_BYTES),
797+
timeoutMilliseconds,
798+
),
799+
);
800+
validateRequestBodyLimitError(
801+
await readJson(
802+
rejectedResponse,
803+
413,
804+
bounds.maxResponseBytes,
805+
"oversized request-body probe",
806+
),
807+
);
808+
809+
const continuationHealthResponse = await fetchImpl(`${baseUrl}/health`, {
810+
headers: { Authorization: authorization },
811+
signal: requestSignal(bounds.clientTimeoutMilliseconds),
812+
});
813+
await expectStatus(
814+
continuationHealthResponse,
815+
200,
816+
bounds.maxResponseBytes,
817+
"request-body continuation health probe",
818+
);
819+
const bodyLimitContinuationResponse = await fetchImpl(
820+
chatUrl,
821+
jsonRequest(
822+
authorization,
823+
{
824+
max_tokens: bounds.maxTokens.synchronousChat,
825+
messages: [{ content: "Return one short continuation token.", role: "user" }],
826+
model,
827+
temperature: 0,
828+
},
829+
timeoutMilliseconds,
830+
),
831+
);
832+
validateChatCompletionResponse(
833+
await readJson(
834+
bodyLimitContinuationResponse,
835+
200,
836+
bounds.maxResponseBytes,
837+
"request-body continuation completion probe",
838+
),
839+
model,
840+
);
841+
executedProbes.add("request-body-limit");
842+
730843
const synchronousResponse = await fetchImpl(
731844
chatUrl,
732845
jsonRequest(
@@ -914,6 +1027,17 @@ export async function runLlamaCppDgxSparkProtocolQualification(options: {
9141027
},
9151028
health: { httpStatus: 200, ok: true },
9161029
malformedRequest: { httpStatus: 400, ok: true },
1030+
requestBodyLimit: {
1031+
acceptedBytes: acceptedRequestBytes,
1032+
acceptedHttpStatus: 200,
1033+
continuationHealthHttpStatus: 200,
1034+
continuationHttpStatus: 200,
1035+
errorCode: "request_body_too_large",
1036+
errorType: "invalid_request_error",
1037+
ok: true,
1038+
rejectedBytes: LLAMA_CPP_DGX_SPARK_REJECTED_REQUEST_BODY_BYTES,
1039+
rejectedHttpStatus: 413,
1040+
},
9171041
models: { httpStatus: 200, model, ok: true },
9181042
metrics,
9191043
properties: propertiesEvidence.properties,

‎scripts/checks/llama-cpp-dgx-spark-qualification-contract.mts‎

Lines changed: 53 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,22 +23,23 @@ export const LLAMA_CPP_DGX_SPARK_MODEL_DIGEST =
2323
"sha256:627f5b04aedc97f967332f331bd75b7a4ed2f33ca83e6ee74b44235cc1887890" as const;
2424
export const LLAMA_CPP_DGX_SPARK_SERVED_MODEL_ID = "nvidia-nemotron-3-nano-30b-a3b" as const;
2525
export const LLAMA_CPP_DGX_SPARK_SOURCE_REVISION =
26-
"22dc605c4ead20e36f447cc67b55ef87e523bd55" as const;
26+
"8e7f22b67ef4667b4ddd50230771287f328cfb3f" as const;
2727
export const LLAMA_CPP_DGX_SPARK_QUALIFICATION_IMAGE_REPOSITORY =
2828
"localhost:5000/nemoclaw-llama-cpp-dgx-spark/llama-cpp-server" as const;
2929
export const LLAMA_CPP_DGX_SPARK_OWNED_IMAGE_REPOSITORY =
3030
"ghcr.io/nvidia/nemoclaw/llama-cpp-server" as const;
3131
export const LLAMA_CPP_DGX_SPARK_SOURCE_REPOSITORY =
3232
"https://github.com/ggml-org/llama.cpp" as const;
3333
export const LLAMA_CPP_DGX_SPARK_SOURCE_ARCHIVE_SHA256 =
34-
"sha256:975f70723e053785e894f4e1d9cf770f2f1a7bc762fd3af174ff5635014108b6" as const;
34+
"sha256:45a24299e7a24410624489d19924d492bc71a120fa17d9b7cb32f6d5c4f1aed0" as const;
3535
export const LLAMA_CPP_DGX_SPARK_CUDA_DEVELOPMENT_BASE =
3636
"docker.io/nvidia/cuda@sha256:ef2203909e80b8b976cfc672f7e2ae2b00bc0e25c404ee86d89e10a3802f1c52" as const;
3737
export const LLAMA_CPP_DGX_SPARK_CUDA_RUNTIME_BASE =
3838
"docker.io/nvidia/cuda@sha256:789e629e49401647e22b7054ae9c6c4f6427dba68010ba428deb4cc6b063676e" as const;
3939
export const LLAMA_CPP_DGX_SPARK_TOOL_IMAGE =
4040
"nvcr.io/nvidia/vllm@sha256:94e21552f644e0c1627464ba89d2f7a4ce7442e196f72afa0bb5d7fba23cbb03" as const;
4141
export const LLAMA_CPP_DGX_SPARK_MINIMUM_DRIVER_VERSION = "580.65.06" as const;
42+
export const LLAMA_CPP_DGX_SPARK_REJECTED_REQUEST_BODY_BYTES = 50_000 as const;
4243
export const LLAMA_CPP_DGX_SPARK_PROTOCOL_PROBES = [
4344
"health",
4445
"models",
@@ -54,6 +55,7 @@ export const LLAMA_CPP_DGX_SPARK_PROTOCOL_PROBES = [
5455
"context-window",
5556
"authentication",
5657
"malformed-request",
58+
"request-body-limit",
5759
"cancellation",
5860
"client-timeout",
5961
] as const;
@@ -469,6 +471,17 @@ export type LlamaCppDgxSparkQualificationReceipt = {
469471
readonly httpStatus: 400;
470472
readonly ok: true;
471473
};
474+
readonly requestBodyLimit: {
475+
readonly acceptedBytes: number;
476+
readonly acceptedHttpStatus: 200;
477+
readonly continuationHealthHttpStatus: 200;
478+
readonly continuationHttpStatus: 200;
479+
readonly errorCode: "request_body_too_large";
480+
readonly errorType: "invalid_request_error";
481+
readonly ok: true;
482+
readonly rejectedBytes: typeof LLAMA_CPP_DGX_SPARK_REJECTED_REQUEST_BODY_BYTES;
483+
readonly rejectedHttpStatus: 413;
484+
};
472485
readonly models: {
473486
readonly httpStatus: 200;
474487
readonly model: typeof LLAMA_CPP_DGX_SPARK_SERVED_MODEL_ID;
@@ -1341,6 +1354,7 @@ export function parseLlamaCppDgxSparkExecutionPlan(
13411354
serve.idleSleepSeconds !== -1 ||
13421355
serve.flashAttention !== "enabled" ||
13431356
serve.speculativeDecoding !== "disabled" ||
1357+
maxRequestBodyBytes !== 32_768 ||
13441358
upstreamPort === serve.port ||
13451359
typeof kvCache.key !== "string" ||
13461360
!allowedKvTypes.has(kvCache.key) ||
@@ -1799,6 +1813,7 @@ export function parseLlamaCppDgxSparkQualificationReceipt(
17991813
"health",
18001814
"logRedaction",
18011815
"malformedRequest",
1816+
"requestBodyLimit",
18021817
"metrics",
18031818
"models",
18041819
"properties",
@@ -1909,6 +1924,22 @@ export function parseLlamaCppDgxSparkQualificationReceipt(
19091924
requireExactKeys(authentication, ["httpStatus", "ok"], "authentication probe");
19101925
const malformedRequest = record(probes.malformedRequest, "malformed-request probe");
19111926
requireExactKeys(malformedRequest, ["httpStatus", "ok"], "malformed-request probe");
1927+
const requestBodyLimit = record(probes.requestBodyLimit, "request-body limit probe");
1928+
requireExactKeys(
1929+
requestBodyLimit,
1930+
[
1931+
"acceptedBytes",
1932+
"acceptedHttpStatus",
1933+
"continuationHealthHttpStatus",
1934+
"continuationHttpStatus",
1935+
"errorCode",
1936+
"errorType",
1937+
"ok",
1938+
"rejectedBytes",
1939+
"rejectedHttpStatus",
1940+
],
1941+
"request-body limit probe",
1942+
);
19121943
const cancellation = record(probes.cancellation, "cancellation probe");
19131944
requireExactKeys(cancellation, ["aborted", "ok", "recovered"], "cancellation probe");
19141945
const clientTimeout = record(probes.clientTimeout, "client-timeout probe");
@@ -1973,6 +2004,15 @@ export function parseLlamaCppDgxSparkQualificationReceipt(
19732004
authentication.httpStatus !== 401 ||
19742005
malformedRequest.ok !== true ||
19752006
malformedRequest.httpStatus !== 400 ||
2007+
requestBodyLimit.ok !== true ||
2008+
requestBodyLimit.acceptedBytes !== expectedPlan.recipe.serve.limits.maxRequestBodyBytes ||
2009+
requestBodyLimit.acceptedHttpStatus !== 200 ||
2010+
requestBodyLimit.rejectedBytes !== LLAMA_CPP_DGX_SPARK_REJECTED_REQUEST_BODY_BYTES ||
2011+
requestBodyLimit.rejectedHttpStatus !== 413 ||
2012+
requestBodyLimit.errorCode !== "request_body_too_large" ||
2013+
requestBodyLimit.errorType !== "invalid_request_error" ||
2014+
requestBodyLimit.continuationHealthHttpStatus !== 200 ||
2015+
requestBodyLimit.continuationHttpStatus !== 200 ||
19762016
cancellation.ok !== true ||
19772017
cancellation.aborted !== true ||
19782018
cancellation.recovered !== true ||
@@ -2049,6 +2089,17 @@ export function parseLlamaCppDgxSparkQualificationReceipt(
20492089
health: { httpStatus: 200, ok: true },
20502090
logRedaction: { ok: true },
20512091
malformedRequest: { httpStatus: 400, ok: true },
2092+
requestBodyLimit: {
2093+
acceptedBytes: expectedPlan.recipe.serve.limits.maxRequestBodyBytes,
2094+
acceptedHttpStatus: 200,
2095+
continuationHealthHttpStatus: 200,
2096+
continuationHttpStatus: 200,
2097+
errorCode: "request_body_too_large",
2098+
errorType: "invalid_request_error",
2099+
ok: true,
2100+
rejectedBytes: LLAMA_CPP_DGX_SPARK_REJECTED_REQUEST_BODY_BYTES,
2101+
rejectedHttpStatus: 413,
2102+
},
20522103
metrics: {
20532104
httpStatus: 200,
20542105
ok: true,

0 commit comments

Comments
 (0)