Local Agentic LLM (Mac M4 Pro 24 GB)

Terminal-first local coding agent on Apple Silicon: OpenCode + Rapid-MLX inference server.

• Privacy-first — models run locally via http://127.0.0.1:8080/v1
• Approval-only — never use --dangerously-skip-permissions
• Hardware: M4 Pro, 24 GB RAM (tuned for this machine)

Default model: mlx-community/Qwen3.5-4B-OptiQ-4bit (~3 GB disk, ~3 GB RAM, 32k context)

---

Architecture

Terminal
├── opencode              # interactive agent (approve each action)
└── terminal only
         │
         ▼
Rapid-MLX server :8080/v1  ←  Qwen3.5 4B OptiQ (Apple Silicon, launchd)

Run OpenCode from your project directory: cd myproject && opencode.

---

Quick start

Prerequisite: Homebrew (brew --version).

git clone <this-repo> ~/local-agentic-llm-setup
cd ~/local-agentic-llm-setup
chmod +x scripts/install.sh scripts/mlx-serve.sh
./scripts/install.sh

First run downloads ~4 GB and may take 10–20 minutes. First inference loads weights into RAM (~30–90s).

---

Commands

Command	Purpose
./scripts/install.sh	Full install — includes verify + HF cache cleanup
./scripts/install.sh --upgrade	Full upgrade + repair (deps, OpenCode, model, verify, cleanup)
./scripts/install.sh --upgrade --best-model	Upgrade and switch to best model for 24 GB if catalog recommends
./scripts/mlx-serve.sh status	Check Rapid-MLX API
./scripts/mlx-serve.sh restart	Restart Rapid-MLX server
./scripts/mlx-serve.sh stop	Stop Rapid-MLX server
opencode	Start terminal agent
terminal only	Use OpenCode TUI directly

Override model: PRIMARY_MODEL=mlx-community/Qwen3-14B-4bit ./scripts/install.sh --upgrade

---

What install does

1. Python venv (.venv) with rapid-mlx and huggingface_hub
2. Downloads the default model from HuggingFace
3. LaunchAgent ai.local.mlx-server on port 8080
4. OpenCode CLI + ~/.config/opencode/opencode.json (Rapid-MLX provider)
5. OPENCODE_ENABLE_EXA=1 in ~/.zshrc (web search)
6. config/models.env — pinned model + versions
7. Removes unused HuggingFace model caches (keeps current model)
8. Verify — stack checks + agent tool test (model reads README.md via tool call)

---

Daily use

source ~/.zshrc                    # once per session (EXA env)
./scripts/mlx-serve.sh status      # API should be "up"
opencode                           # approve edits and shell commands

Use OpenCode TUI for terminal interaction.

Switch model in session: /models in the OpenCode TUI.

---

Model

HuggingFace ID	mlx-community/Qwen3.5-4B-OptiQ-4bit
Quantization	uniform 4-bit

Why this model: Best tool-calling reliability (F1=0.919), fast, fits comfortably in 24 GB.

Recommended model for 24 GB Mac:

Model	RAM	Notes
mlx-community/Qwen3.5-4B-OptiQ-4bit	~3 GB	Default — best tool-calling

Server defaults (edit scripts/mlx-serve.sh if needed): port 8080, max tokens 16384, thinking mode off, prefix caching on, auto tool-call parser.

Speed tips: keep launchd server running; fresh OpenCode session for long tasks; disable unused MCPs; optional pre-warm: curl http://127.0.0.1:8080/v1/models.

Weights cache: ~/.cache/huggingface/hub/ (old models pruned automatically on install/upgrade).

Every ./scripts/install.sh --upgrade runs a model check against HuggingFace and config/recommended-models.json (ranked for M4 Pro 24 GB agent use):

• Current model — Hub revision date and whether your pinned digest is stale (same model id, newer weights)
• Recommended — best catalog entry that fits the 6 GB RAM budget (Qwen3.5-4B-OptiQ-4bit is the default pick today)
• Watch — polls Hub for models in watch inside config/recommended-models.json; flags the moment either lands and auto-recommends it when available
• New on Hub — newer Qwen models not yet in the catalog (hint to update rankings)

If output says a better model is available:

./scripts/install.sh --upgrade --best-model

That switches PRIMARY_MODEL, downloads weights, and re-applies the stack. To pin a specific model manually: PRIMARY_MODEL=mlx-community/... ./scripts/install.sh --upgrade.

Edit rankings in config/recommended-models.json when you want to adopt new Hub models.

---

Upgrade

./scripts/install.sh --upgrade

One command upgrades Python deps, OpenCode, model weights, restarts the server, re-applies OpenCode config, cleans unused HF caches, runs a model check, and verify. Output shows Before, per-component (unchanged) or before → after, then After.

If anything breaks (wrong provider, stale config, server down): run --upgrade again.

---

OpenCode config

Written by install.sh / --upgrade:

• Provider: mlx → http://127.0.0.1:8080/v1
• Agent: build, tool_call: true
• Permissions: read/grep/websearch allowed; edit/bash/write require approval
• Timeout: 10 min per turn

Optional MCP stubs (GitHub, Context7, Sentry) in opencode.json.example — all disabled by default. Fewer MCPs = faster turns.

Project-level opencode.json (optional) merges with global config — e.g. Foundry MCP stub in this repo.

---

Troubleshooting

Problem	Fix
Rapid-MLX API down	./scripts/mlx-serve.sh status → restart or ./scripts/install.sh --upgrade
OpenCode "cannot connect"	Server not up yet — wait 30–90s after restart, or ./scripts/mlx-serve.sh status
Wrong model / config	./scripts/install.sh --upgrade
Slow first prompt	Normal — cold load into RAM
Out of memory	PRIMARY_MODEL=mlx-community/Qwen3.5-4B-OptiQ-4bit ./scripts/install.sh --upgrade
HuggingFace rate limits	export HF_TOKEN=hf_... then --upgrade
pip / venv broken	rm -rf .venv && ./scripts/install.sh

Logs: ./scripts/mlx-serve.sh logs

---

Success checklist

• ./scripts/install.sh completes (includes verify)
• ./scripts/mlx-serve.sh status → API up, model listed
• opencode connects and uses tools (read files, run approved commands)