Skip to content

ollama.md

Latest commit

 

History

History
188 lines (126 loc) · 7.91 KB

ollama.md

File metadata and controls

188 lines (126 loc) · 7.91 KB
Raw
layout page
title Ollama: Add Models
description How to add local Ollama models to NodeTool and how to change provider-level behavior such as tool-support detection.

The Ollama provider connects NodeTool to a local Ollama daemon. Models are discovered dynamically from the running daemon — no code change is needed to add a new model. Provider-level changes (tool detection, embedding behavior, URL configuration) require editing the provider source.

Audience: coding agents and contributors — both users adding local models and devs modifying the provider.

TL;DR

ollama pull llama3.2          # makes the model appear in NodeTool automatically
ollama pull nomic-embed-text  # embedding models work the same way

No code change. No restart required (NodeTool queries the daemon on every model-list call). If you need to change tool-support detection, the default URL, or keep_alive behavior, see Provider-level changes below.

Where things live

Concern Path
Provider implementation packages/runtime/src/providers/ollama-provider.ts
Default daemon URL packages/runtime/src/providers/defaults.ts (OLLAMA_DEFAULT_URL)
Registration + optionalKwargs packages/runtime/src/providers/index.ts (lines 248–253)
Provider registry resolution logic packages/runtime/src/providers/provider-registry.ts

How Ollama models are discovered

OllamaProvider.getAvailableLanguageModels() (line 357 of ollama-provider.ts) fetches GET {apiUrl}/api/tags from the local daemon on every call. It maps each entry's model field (falling back to name) to a LanguageModel record:

// packages/runtime/src/providers/ollama-provider.ts:357-372
async getAvailableLanguageModels(): Promise<LanguageModel[]> {
  const response = await this._fetch(`${this.apiUrl}/api/tags`);
  if (!response.ok) return [];
  const payload = (await response.json()) as {
    models?: Array<{ name?: string; model?: string }>;
  };
  ...
}

getAvailableEmbeddingModels() (line 374) calls getAvailableLanguageModels() and re-wraps every result as an EmbeddingModel with dimensions: 0. Ollama does not advertise context size or dimensions over /api/tags, so both methods surface all pulled models regardless of type. The caller (RAG pipeline, agent config) is responsible for picking an appropriate model.

URL resolution order (highest priority first):

  1. Secret store — key OLLAMA_API_URL (set via nodetool secrets store OLLAMA_API_URL or Settings → API Keys)
  2. Environment variable — OLLAMA_API_URL
  3. Registered default — http://127.0.0.1:11434 (OLLAMA_DEFAULT_URL in defaults.ts)

The registration in index.ts uses an optionalKwarg for OLLAMA_API_URL so the registry always considers Ollama "configured" (no key required) while still resolving a user-set URL on every getProvider() call without a restart.

// packages/runtime/src/providers/index.ts:248-253
registerBuiltinProvider(
  PROVIDER_IDS.OLLAMA,
  OllamaProvider,
  {},
  { OLLAMA_API_URL: OLLAMA_DEFAULT_URL }
);

Ollama is registered only when NODETOOL_ENV !== "production" (lines 241–282 of index.ts). In production/cloud deployments the provider is pruned from the registry.

The constructor also reads OLLAMA_KEEP_ALIVE from process.env. Default is "10m". Set it to "-1" to keep models resident indefinitely, or "0" to unload immediately after each request.

Add a model (user path)

Pull the model with the Ollama CLI:

# Language models
ollama pull llama3.2
ollama pull gemma3:27b
ollama pull phi4
ollama pull deepseek-r1:8b

# Embedding models
ollama pull nomic-embed-text
ollama pull mxbai-embed-large

# Vision models (multimodal)
ollama pull llava:13b
ollama pull minicpm-v

NodeTool picks up the model on the next call to getAvailableLanguageModels() or getAvailableEmbeddingModels() — no restart needed. The model ID passed to the provider must match the tag exactly as shown by ollama list (e.g. llama3.2:latest, gemma3:27b).

To point NodeTool at a non-default Ollama instance (remote host, different port):

npm run dev:nodetool -- secrets store OLLAMA_API_URL
# Paste the URL when prompted, e.g.: http://192.168.1.10:11434

Or set the environment variable before starting the server:

OLLAMA_API_URL=http://192.168.1.10:11434 npm run dev

Provider-level changes (dev path)

Edit packages/runtime/src/providers/ollama-provider.ts. The main extension points:

Tool-support detection

hasToolSupport(model) (line 129) queries POST {apiUrl}/api/show and checks for "tools" in the capabilities array. It caches results in _modelInfoCache (a Map<string, Record<string, unknown>>). When the /api/show call fails or the model has no capabilities field, it defaults to true.

If a model reports no tool support, generateMessage and generateMessages fall back to _injectToolEmulationPrompt + _parseEmulatedToolCalls, which inject tool descriptions into the system prompt and parse function_name(param='value') patterns from the output.

To override detection for a specific model (e.g. a model that advertises tools but calls them incorrectly), add a guard at the top of hasToolSupport:

if (model.startsWith("my-broken-model")) return false;

Embedding dimensions

getAvailableEmbeddingModels() returns dimensions: 0 for every model because Ollama's /api/tags response does not include that field. If you need accurate dimensions, query POST /api/show for the pulled model and read model_info["embedding_length"]. The _modelInfoCache already stores the full /api/show response, so the call is free on the second access.

Keep-alive

Change the default (currently "10m") by editing the fallback in the constructor:

// packages/runtime/src/providers/ollama-provider.ts:109-110
const keepAlive = process.env.OLLAMA_KEEP_ALIVE?.trim();
this.keepAlive = keepAlive && keepAlive.length > 0 ? keepAlive : "10m";

Default URL

Change OLLAMA_DEFAULT_URL in packages/runtime/src/providers/defaults.ts. The constant is re-exported from packages/runtime/src/providers/index.ts so callers import it as @nodetool-ai/runtime.

Verify

# 1. Confirm the daemon is running and a model is present
curl http://127.0.0.1:11434/api/tags

# 2. Type-check the provider package
npm run lint --workspace=packages/runtime

# 3. Run a single Ollama-backed node (replace model with one you have pulled)
npm run dev:nodetool -- node run nodetool.agents.Agent \
  --props '{"prompt":"hello","model":{"type":"language_model","provider":"ollama","id":"llama3.2:latest"}}'

# 4. Full suite
npm run check

If the daemon is not running, getAvailableLanguageModels() returns [] silently (the provider guards if (!response.ok) return []). Check that ollama serve (or the Ollama desktop app) is running before debugging further.

How past commits did it

The provider file was first tracked in commit 4469fd80 ("Add TypeScript backend packages from nodetool-core/ts"), which established the provider registration pattern. The two mechanics the Ollama registration leans on arrived later: the gate for local-only providers in b0e0b8bc ("feat: gate local-only features behind NODETOOL_ENV=production"), and the optionalKwargs slot that resolves OLLAMA_API_URL on every getProvider() call in 532a53b5.

The registration block in packages/runtime/src/providers/index.ts still passes {} for required kwargs, putting OLLAMA_API_URL in the optional slot with OLLAMA_DEFAULT_URL as its fallback — the zero-required-kwarg pattern this page describes.

Contributing

PRs welcome at https://github.com/nodetool-ai/nodetool. Before pushing:

npm run lint        # must pass
npm run typecheck   # must pass

Join the discussion on Discord.