You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
`^Invalid port: ${PORT_ENV_NAME}="\\d{1,5}" — (?:must be an integer between 1024 and 65535|must not overlap the 18789-18799 dashboard port range|must not overlap the (?:llama\\.cpp inference|vLLM / NIM inference|Ollama inference|Ollama auth proxy|Bedrock Runtime adapter|OpenRouter Runtime adapter|HTTPS Pin Runtime adapter) default port \\(\\d{1,5}\\)|conflicts with ${PORT_ENV_NAME} \\(\\d{1,5}\\))$`,
22
+
`^Invalid port: ${PORT_ENV_NAME}="\\d{1,5}" — (?:must be an integer between 1024 and 65535|must not overlap the 18789-18799 dashboard port range|must not overlap the (?:llama\\.cpp inference|vLLM / NIM inference|Ollama inference|Ollama auth proxy|Bedrock Runtime adapter|OpenRouter Runtime adapter|HTTPS Pin Runtime adapter) default port \\(\\d{1,5}\\)|conflicts with ${PORT_ENV_NAME} \\(\\d{1,5}\\)|conflicts with the fixed llama\\.cpp inference port \\(8081\\))$`,
The flag enables the dedicated vLLM profile gate, disables Express selection, and selects the matching catalog recipe.
114
114
Selecting Express option 2 enables the same profile gate and selects the vLLM runtime.
115
115
Do not combine this path with `NEMOCLAW_PROVIDER` or `NEMOCLAW_MODEL`.
116
-
The profile rejects `NEMOCLAW_VLLM_MODEL` and `NEMOCLAW_VLLM_EXTRA_ARGS_JSON` before installation.
117
-
Set `NEMOCLAW_VLLM_PORT` to publish its fixed container listener on another unused host port.
116
+
The profile rejects `NEMOCLAW_VLLM_MODEL`, `NEMOCLAW_VLLM_PORT`, and `NEMOCLAW_VLLM_EXTRA_ARGS_JSON` before installation.
118
117
119
118
The profile performs these actions:
120
119
121
120
- Uses the pinned runtime image and fixed serving command from the catalog.
122
121
- Stores model files in the host Hugging Face cache.
123
-
- Publishes the authenticated server on loopback and the exact private IPv4 gateway of the `openshell-docker` bridge at the configured `NEMOCLAW_VLLM_PORT` (`8000` by default).
122
+
- Publishes the authenticated server on `127.0.0.1:8000` and the exact private IPv4 gateway of the `openshell-docker` bridge, such as `172.18.0.1:8000`.
124
123
- Never publishes the Docker host port on `0.0.0.0` or `::`.
125
124
- Reuses the owner-only host-global managed-vLLM API key.
126
125
@@ -149,7 +148,7 @@ docker container inspect \
149
148
```
150
149
151
150
The bridge result must contain exactly one private IPv4 `Gateway` value.
152
-
The container result must contain exactly two bindings with `HostPort` set to the selected `NEMOCLAW_VLLM_PORT` (`8000` by default): one `HostIp` value of `127.0.0.1` and one that exactly matches the private bridge gateway. The inspect key remains `8000/tcp` because the container listener is fixed even when the published host port changes.
151
+
The container result must contain exactly two bindings with `HostPort` set to `8000`: one `HostIp` value of `127.0.0.1` and one that exactly matches the private bridge gateway.
153
152
Reject the runtime if a binding uses `0.0.0.0`, `::`, an empty address, another host address, or an additional entry.
154
153
155
154
If installation stops after a model download, rerun the same installer command.
Copy file name to clipboardExpand all lines: docs/inference/set-up-llama-cpp.mdx
+11-18Lines changed: 11 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,7 +17,7 @@ Both paths are experimental and use OpenAI Chat Completions through `https://inf
17
17
18
18
| Path | NemoClaw manages | You manage | Use it when |
19
19
|---|---|---|---|
20
-
| Existing server | Provider registration, route validation, and sandbox traffic through `inference.local`. | The server binary or container, GGUF file, launch configuration, credential, upgrades, and lifecycle. | Your authenticated server already satisfies the llama.cpp fingerprint contract on the configured loopback port (`8081` by default). |
20
+
| Existing server | Provider registration, route validation, and sandbox traffic through `inference.local`. | The server binary or container, GGUF file, launch configuration, credential, upgrades, and lifecycle. | Your authenticated server already satisfies the llama.cpp fingerprint contract on loopback port `8081`. |
21
21
| Managed DGX Spark | The exact YAML-selected image, GGUF acquisition and verification, launch configuration, credential, Docker lifecycle, diagnostics, and cleanup. | The qualified Linux host, Docker Engine, NVIDIA runtime prerequisites, storage, and any required Hugging Face credential. | You want the experimental Muse Glimmer or NVIDIA Nemotron profile without managing `llama-server` directly. |
22
22
23
23
Compared with other local options, Ollama emphasizes a simple local model workflow, while managed vLLM provides host-specific NVIDIA GPU profiles and broader model selection.
@@ -30,7 +30,7 @@ Refer to [Choose a Local Inference Server](choose-local-inference-server) for th
30
30
Use this path when you operate `llama-server` and want NemoClaw to register its authenticated endpoint without owning its process or model.
31
31
The server must satisfy all of these requirements:
32
32
33
-
- Listen over HTTP on `127.0.0.1:<configured-port>` and expose its OpenAI-compatible API under `/v1`.
33
+
- Listen over HTTP on `127.0.0.1:8081` and expose its OpenAI-compatible API under `/v1`.
34
34
- Require the same native bearer key supplied through `NEMOCLAW_LLAMACPP_LOCAL_TOKEN`.
35
35
- Expose bounded native llama.cpp evidence through `/v1/models`, `/health`, `/props`, and either `/metrics` or the native metrics-not-supported response.
36
36
- Report a stable, non-path served model alias.
@@ -64,7 +64,6 @@ For non-interactive onboarding, export `NEMOCLAW_LLAMACPP_LOCAL_TOKEN` from your
64
64
:"${NEMOCLAW_LLAMACPP_LOCAL_TOKEN:?Export the native llama.cpp API key first}"
@@ -121,7 +116,7 @@ Run `unset HF_TOKEN` after onboarding when no other process needs it.
121
116
<Warning>
122
117
When the exact artifacts are not already present, onboarding pulls digest-pinned images from external registries and acquires the YAML-declared GGUF file through the standard Hugging Face cache.
123
118
It reuses locally present pinned images and a cached GGUF only after their exact identities verify.
124
-
It creates an owner-only API key, a host-loopback listener on the configured port, and a Docker internal network.
119
+
It creates an owner-only API key, a host-loopback listener on port`8081`, and a Docker internal network.
125
120
Only one managed llama.cpp runtime can exist per Docker authority, regardless of the owning OpenShell gateway or sandbox.
126
121
</Warning>
127
122
@@ -143,7 +138,6 @@ For non-interactive onboarding, select the repository-owned recipe explicitly:
@@ -157,8 +151,7 @@ For non-interactive onboarding or an explicit provider request, set `NEMOCLAW_LL
157
151
An unknown recipe, an incompatible host, or more than one compatible profile for the requested recipe stops onboarding before installation effects.
158
152
Do not set `NEMOCLAW_MODEL` for this path.
159
153
The repository YAML recipe is authoritative for the exact model file, revision, digest, image, launch settings, resource limits, readiness probes, and disabled surfaces.
160
-
`NEMOCLAW_LLAMACPP_PORT` controls the host-owned private bridge; the recipe's internal server port remains fixed.
161
-
NemoClaw does not provide hidden model, image, or launch-flag overrides for the managed path.
154
+
NemoClaw does not provide hidden model, port, image, or launch-flag overrides for the managed path.
162
155
163
156
The shipped NVIDIA Nemotron recipe continues to use the `nemotron-v3-embedded` chat template.
164
157
A repository recipe that uses an image-owned Jinja template must declare this typed contract:
@@ -196,7 +189,7 @@ Onboarding performs these actions:
196
189
- Reuses a verified GGUF from the shared `~/.cache/huggingface/` cache, or acquires the exact file through the existing Hugging Face mechanism when it is absent or invalid.
197
190
- Verifies the GGUF size and SHA-256 digest before it starts the runtime.
198
191
- Runs the authenticated container as the current non-root user with one NVIDIA GPU and no CPU fallback.
199
-
- Starts a host-owned private bridge on `127.0.0.1:<configured-port>` and connects the container to a Docker internal network without a Docker-published port.
192
+
- Starts a host-owned private bridge on `127.0.0.1:8081` and connects the container to a Docker internal network without a Docker-published port.
200
193
- Registers `llama-cpp-local` and routes agent traffic through `https://inference.local/v1`.
201
194
202
195
The runtime cannot download a model and has no egress on its internal Docker network.
@@ -232,14 +225,14 @@ Refer to [CLI Commands](../../reference/commands) for complete command behavior.
232
225
233
226
During managed installation, NemoClaw first checks internal runtime readiness and host-loopback health.
234
227
It repeats these checks when onboarding resumes.
235
-
It then runs the authoritative OpenShell Docker bridge probe on the configured host port.
228
+
It then runs the authoritative OpenShell Docker bridge probe on fixed port `8081`.
236
229
If that bridge probe cannot connect or times out after the earlier checks pass, onboarding stops without changing UFW.
237
230
When the bridge reports a valid narrow subnet and a gateway IP address inside that subnet, the error reports these values:
238
231
239
232
- The detected Docker network.
240
233
- The source subnet.
241
234
- The gateway IP address.
242
-
- The configured port.
235
+
- The fixed port `8081`.
243
236
- The exact narrow UFW command.
244
237
245
238
If the bridge topology is missing or invalid, onboarding fails closed without printing topology or a UFW command.
@@ -248,22 +241,22 @@ Inspect the OpenShell Docker bridge configuration before you retry onboarding.
248
241
<Warning>
249
242
The printed command changes UFW.
250
243
Confirm that the reported Docker network, source subnet, and gateway IP address belong to the OpenShell Docker bridge before you run it.
251
-
The rule allows TCP traffic only from that subnet to the configured port on that gateway IP address.
244
+
The rule allows TCP traffic only from that subnet to port `8081` on that gateway IP address.
252
245
</Warning>
253
246
254
247
Run the exact command from the onboarding error.
255
248
It has this form:
256
249
257
250
```bash
258
-
sudo ufw allow from <subnet> to <gateway-ip> port <llama-cpp-port> proto tcp
251
+
sudo ufw allow from <subnet> to <gateway-ip> port 8081 proto tcp
259
252
```
260
253
261
254
After you apply the rule, rerun the same onboarding selection.
262
255
263
256
If onboarding still reports the bridge failure, or when the managed runtime no longer needs the rule, remove the exact rule that you added:
264
257
265
258
```bash
266
-
sudo ufw --force delete allow from <subnet> to <gateway-ip> port <llama-cpp-port> proto tcp
259
+
sudo ufw --force delete allow from <subnet> to <gateway-ip> port 8081 proto tcp
267
260
```
268
261
269
262
Use the same subnet and gateway IP address that appeared in the original command.
0 commit comments