Skip to content

Commit 00aa0b8

Browse files
HOYALIMHOYALIM
committed
docs: add model capability audit matrix
Signed-off-by: Ho Lim <subhoya@gmail.com>
1 parent 5d115f8 commit 00aa0b8

3 files changed

Lines changed: 194 additions & 0 deletions

File tree

‎docs/index.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,9 @@ navigation:
6363
- page: "Tool-Calling Reliability"
6464
path: inference/tool-calling-reliability.mdx
6565
slug: tool-calling-reliability
66+
- page: "Model Capability Audit"
67+
path: _build/agent-variants/inference/model-capability-audit.openclaw.generated.mdx
68+
slug: model-capability-audit
6669
- page: "Switch Inference Providers"
6770
path: _build/agent-variants/inference/switch-inference-providers.openclaw.generated.mdx
6871
slug: switch-inference-providers
@@ -234,6 +237,9 @@ navigation:
234237
- page: "Use Local Inference"
235238
path: _build/agent-variants/inference/use-local-inference.hermes.generated.mdx
236239
slug: use-local-inference
240+
- page: "Model Capability Audit"
241+
path: _build/agent-variants/inference/model-capability-audit.hermes.generated.mdx
242+
slug: model-capability-audit
237243
- page: "Switch Inference Providers"
238244
path: _build/agent-variants/inference/switch-inference-providers.hermes.generated.mdx
239245
slug: switch-inference-providers

‎docs/inference/model-capability-audit.mdx‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
---
2+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
# SPDX-License-Identifier: Apache-2.0
4+
title: "Model Capability Audit Matrix"
5+
sidebar-title: "Model Capability Audit"
6+
description: "Maintained matrix template for auditing NemoClaw model and provider behavior across supported agent surfaces."
7+
description-agent: "Defines the maintained model capability audit matrix schema, states, evidence requirements, and seed rows. Use when adding or reviewing model/provider compatibility audit evidence."
8+
keywords: ["nemoclaw model audit", "model capability matrix", "provider compatibility audit", "agent model validation"]
9+
topics: ["inference", "model validation", "provider compatibility"]
10+
tags: ["model audit", "agent validation", "inference providers"]
11+
content:
12+
type: "reference"
13+
difficulty: "intermediate"
14+
audience: ["maintainers", "contributors"]
15+
status: "maintained"
16+
exclude-from-skills-gen: true
17+
---
18+
Use this matrix to maintain model and provider audit evidence for NemoClaw agent behavior.
19+
The matrix tracks whether a supported model works as an agent model, not only whether it can answer a one-shot chat prompt.
20+
21+
Do not mark a row as completed without committed evidence or a stable CI link.
22+
Rows seeded from source inventory start as `not-yet-run` until a maintainer imports or records evidence.
23+
24+
## Result States
25+
26+
Every audit row must use one of these states.
27+
28+
| State | Use when |
29+
|---|---|
30+
| `pass` | The row completes required scenarios without model-specific changes. |
31+
| `pass-with-affordance` | The row completes required scenarios with a documented model or provider affordance. |
32+
| `degraded` | The row is usable but has documented limits, retries, latency risk, or partial surface coverage. |
33+
| `blocked` | The row cannot complete required scenarios and needs a linked follow-up issue or PR. |
34+
| `unsupported` | The model, provider, or surface is intentionally unsupported. |
35+
| `not-yet-run` | The row is in scope but has no completed evidence yet. |
36+
37+
## Required Row Schema
38+
39+
Use these fields for every completed row.
40+
If a field is not applicable, write `n/a` and explain why in the evidence notes.
41+
42+
| Field | Required content |
43+
|---|---|
44+
| Model ID | Exact model identifier used by onboarding or runtime config. |
45+
| Provider path | Provider class and route, such as NVIDIA Endpoints, OpenAI, Anthropic, Gemini, Local Ollama, Local vLLM, or another compatible endpoint. |
46+
| Agent surface | Exact agent path, such as OpenClaw primary agent, OpenClaw CLI prompt path, OpenClaw browser or gateway path, OpenClaw sub-agent delegation, Hermes sandbox API, or auxiliary model path. |
47+
| NemoClaw commit SHA | Full commit SHA for the repo state used during validation. |
48+
| Runtime versions | OpenShell, OpenClaw, Hermes, provider server, and local serving versions when available. |
49+
| Endpoint/API path selected | Concrete API path, base URL class, and provider key selected by NemoClaw. |
50+
| Workflow used | Exact command sequence or CI workflow used to run the scenario. |
51+
| State | One result state from this page. |
52+
| Evidence | Trajectory file path, session log path, request dump path, or CI artifact link. |
53+
| Observed tool-call count | Count and names of structured tool calls observed in the scenario. |
54+
| Final-response behavior | Whether the assistant produced a final response after tool results, stopped empty, stopped reasoning-only, or emitted raw tool text. |
55+
| Multi-turn behavior | Whether turn 2 used turn 1 tool results without re-running unrelated tools. |
56+
| Latency and timeout notes | Validation time, first token or first event time when available, total duration, retries, and timeout budget used. |
57+
| Required affordance | Model-specific setup, provider-class transport behavior, request mutation, API path forcing, streaming requirement, or `none`. |
58+
| Follow-up | Linked issue, PR, or registry decision when remediation or setup work is needed. |
59+
60+
## Required Scenario Coverage
61+
62+
Completed rows should state which required scenarios were exercised.
63+
Rows can remain `degraded`, `blocked`, or `not-yet-run` when a scenario cannot be exercised yet.
64+
65+
| Scenario | Required checks |
66+
|---|---|
67+
| Baseline chat | Deterministic response works, provider validation is actionable, and credentials do not leak into sandbox-visible files, logs, or prompts. |
68+
| Shell tool loop | Separate structured `hostname`, `date`, and `uptime` tool calls are emitted, persisted, correlated with tool results, and followed by a final assistant response. |
69+
| Multi-turn continuation | Turn 2 uses a tool result from turn 1 and does not ask the user to continue after a complete tool result. |
70+
| Sub-agent delegation | The primary agent emits a structured `sessions_spawn` request, the sub-agent receives the intended task and workspace, and the primary agent consumes the result. |
71+
| Hermes path | Hermes starts with the selected provider/model, returns the expected OpenAI-compatible response shape, and separates Hermes failures from OpenClaw-only request-shape issues. |
72+
| Performance and operability | The row records validation duration, first event timing when available, retry behavior, timeout budget, streaming requirement, request mutation requirement, API path forcing, and cold-start differences. |
73+
74+
## Audit Matrix
75+
76+
These seed rows come from current repo source files, not from live benchmark claims.
77+
Keep them as `not-yet-run` until the row has evidence that satisfies the schema above.
78+
When importing a completed row from an issue comment, preserve the exact commit SHA, workflow, evidence paths, and observed behavior.
79+
80+
| Agent surface | Provider class | Model or route | API path | State | Evidence | Required affordance | Follow-up | Source |
81+
|---|---|---|---|---|---|---|---|---|
82+
| OpenClaw primary agent | NVIDIA Endpoints | `nvidia/nemotron-3-super-120b-a12b` | Managed `inference.local` OpenAI-compatible completions | `not-yet-run` | Add trajectory and session evidence before changing state. | Existing OpenClaw setup manifest disables `tool_search` for this route. | Verify evidence before changing state. | `src/lib/inference/config.ts`, `nemoclaw-blueprint/model-specific-setup/openclaw/nemotron-3-super-120b-managed-inference.json`. |
83+
| OpenClaw primary agent | NVIDIA Endpoints | `moonshotai/kimi-k2.6` | Managed `inference.local` OpenAI-compatible completions | `not-yet-run` | Add trajectory and session evidence before changing state. | Existing OpenClaw setup manifest applies Kimi compatibility and plugin loading. | Verify Kimi regression evidence before changing state. | `src/lib/inference/config.ts`, `nemoclaw-blueprint/model-specific-setup/openclaw/kimi-k2.6-managed-inference.json`. |
84+
| OpenClaw primary agent | NVIDIA Endpoints | Any model from `CLOUD_MODEL_OPTIONS` | Managed `inference.local` OpenAI-compatible completions unless config selects another API. | `not-yet-run` | Add one evidence row per model before changing state. | Record `none`, model-specific setup, or provider-class transport behavior. | Expand into per-model rows as evidence lands. | `src/lib/inference/config.ts`. |
85+
| OpenClaw primary agent | OpenAI | Any model from `REMOTE_MODEL_OPTIONS.openai` | `openai` provider through `https://inference.local/v1`. | `not-yet-run` | Add one evidence row per model before changing state. | Record Responses or Chat Completions behavior explicitly. | Expand into per-model rows as evidence lands. | `src/lib/inference/model-prompts.ts`, `src/lib/inference/config.ts`. |
86+
| OpenClaw primary agent | Anthropic | Any model from `REMOTE_MODEL_OPTIONS.anthropic` | `anthropic` provider through `https://inference.local` with `anthropic-messages`. | `not-yet-run` | Add one evidence row per model before changing state. | Record native Anthropic Messages behavior explicitly. | Expand into per-model rows as evidence lands. | `src/lib/inference/model-prompts.ts`, `src/lib/inference/config.ts`. |
87+
| OpenClaw primary agent | Gemini | Any model from `REMOTE_MODEL_OPTIONS.gemini` | Managed `inference.local` OpenAI-compatible route. | `not-yet-run` | Add one evidence row per model before changing state. | Record provider state and tool-result continuation behavior. | Expand into per-model rows as evidence lands. | `src/lib/inference/model-prompts.ts`, `src/lib/inference/config.ts`. |
88+
| OpenClaw primary agent | Local Ollama | Default `nemotron-3-nano:30b` or any installed model selected by onboarding. | Managed `inference.local` route to the host Ollama proxy. | `not-yet-run` | Add local daemon, model tag, and trajectory evidence before changing state. | Record tool capability, streaming usage, and local proxy behavior. | Add one row per audited local model tag. | `src/lib/inference/local.ts`, `src/lib/inference/config.ts`. |
89+
| OpenClaw primary agent | Local vLLM | Any model from `VLLM_MODELS`. | Managed `inference.local` route to the host vLLM server. | `not-yet-run` | Add vLLM serve flags, model id, and trajectory evidence before changing state. | Record parser flags, reasoning parser, and tool-call parser behavior. | Add one row per audited vLLM model id. | `src/lib/inference/vllm-models.ts`, `src/lib/inference/config.ts`. |
90+
| OpenClaw primary agent | Other OpenAI-compatible endpoint | User-selected `custom-model` or another configured model id. | Managed `inference.local` route to the compatible endpoint. | `not-yet-run` | Add endpoint class and trajectory evidence before changing state. | Record endpoint API path forcing and store/streaming assumptions. | Add one row per endpoint class that is validated. | `src/lib/inference/config.ts`. |
91+
| OpenClaw primary agent | Other Anthropic-compatible endpoint | User-selected `custom-anthropic-model` or another configured model id. | `anthropic` route when supported, otherwise managed compatible route. | `not-yet-run` | Add endpoint class and trajectory evidence before changing state. | Record native Anthropic Messages or compatible-route transport behavior. | Add one row per endpoint class that is validated. | `src/lib/inference/config.ts`. |
92+
| Hermes sandbox API | Hermes Provider | Default `moonshotai/kimi-k2.6` or any model from `HERMES_PROVIDER_MODEL_OPTIONS`. | Hermes Provider route through NemoClaw managed inference. | `not-yet-run` | Add Hermes session, request dump, logs, and local API evidence before changing state. | Record Hermes-specific config, transport, and response-shape behavior. | Keep Hermes rows separate from OpenClaw rows. | `src/lib/inference/config.ts`, `src/lib/inference/model-prompts.ts`. |
93+
94+
## Completed Row Template
95+
96+
Copy this template when adding evidence for a specific model/provider/agent combination.
97+
Do not leave placeholder text in a completed row.
98+
99+
| Field | Value |
100+
|---|---|
101+
| Model ID | `<provider/model-id>`. |
102+
| Provider path | `<provider class and route>`. |
103+
| Agent surface | `<OpenClaw primary agent, OpenClaw CLI prompt path, OpenClaw browser or gateway path, OpenClaw sub-agent delegation, Hermes sandbox API, or auxiliary model path>`. |
104+
| NemoClaw commit SHA | `<full SHA>`. |
105+
| Runtime versions | `<OpenShell version, OpenClaw version, Hermes version, local server version, or n/a>`. |
106+
| Endpoint/API path selected | `<provider key, base URL class, API mode, and endpoint path>`. |
107+
| Workflow used | `<exact commands or CI workflow>`. |
108+
| State | `<pass, pass-with-affordance, degraded, blocked, unsupported, or not-yet-run>`. |
109+
| Evidence | `<trajectory, session log, request dump, CI artifact, or n/a>`. |
110+
| Observed tool-call count | `<count, names, and shape>`. |
111+
| Final-response behavior | `<final answer, empty stop, reasoning-only stop, raw tool text, or other behavior>`. |
112+
| Multi-turn behavior | `<turn 1 and turn 2 behavior>`. |
113+
| Latency and timeout notes | `<validation time, first event timing, total duration, retry behavior, timeout budget, and streaming notes>`. |
114+
| Required affordance | `<none, setup manifest, request mutation, parser flag, API path forcing, streaming requirement, or transport policy>`. |
115+
| Follow-up | `<issue, PR, registry decision, or n/a>`. |
116+
117+
## Related Artifacts
118+
119+
- `nemoclaw-blueprint/model-specific-setup/README.md` documents where model-specific setup belongs once an intervention is justified.
120+
- `docs/inference/tool-calling-reliability` explains the local inference tool-call failure mode that audit rows should classify separately from provider connectivity.

‎test/model-capability-audit-doc.test.ts‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
import fs from "node:fs";
5+
import path from "node:path";
6+
import { fileURLToPath } from "node:url";
7+
import { describe, expect, it } from "vitest";
8+
9+
const __filename = fileURLToPath(import.meta.url);
10+
const __dirname = path.dirname(__filename);
11+
const repoRoot = path.resolve(__dirname, "..");
12+
const auditDocPath = path.join(repoRoot, "docs", "inference", "model-capability-audit.mdx");
13+
const navPath = path.join(repoRoot, "docs", "index.yml");
14+
15+
const resultStates = [
16+
"pass",
17+
"pass-with-affordance",
18+
"degraded",
19+
"blocked",
20+
"unsupported",
21+
"not-yet-run",
22+
] as const;
23+
24+
const evidenceFields = [
25+
"Model ID",
26+
"Provider path",
27+
"Agent surface",
28+
"NemoClaw commit SHA",
29+
"Runtime versions",
30+
"Endpoint/API path selected",
31+
"Workflow used",
32+
"State",
33+
"Evidence",
34+
"Observed tool-call count",
35+
"Final-response behavior",
36+
"Multi-turn behavior",
37+
"Latency and timeout notes",
38+
"Required affordance",
39+
"Follow-up",
40+
] as const;
41+
42+
describe("model capability audit doc (#3123)", () => {
43+
it("keeps the maintained audit states and evidence schema", () => {
44+
const markdown = fs.readFileSync(auditDocPath, "utf8");
45+
46+
for (const state of resultStates) {
47+
expect(markdown).toContain(`\`${state}\``);
48+
}
49+
for (const field of evidenceFields) {
50+
expect(markdown).toContain(field);
51+
}
52+
53+
expect(markdown).toContain(
54+
"Agent surface | Provider class | Model or route | API path | State | Evidence",
55+
);
56+
});
57+
58+
it("links the audit page from both guide variants", () => {
59+
const nav = fs.readFileSync(navPath, "utf8");
60+
61+
expect(nav).toContain(
62+
"_build/agent-variants/inference/model-capability-audit.openclaw.generated.mdx",
63+
);
64+
expect(nav).toContain(
65+
"_build/agent-variants/inference/model-capability-audit.hermes.generated.mdx",
66+
);
67+
});
68+
});

0 commit comments

Comments
 (0)