feat(llm, ui): Add native keyless Local LLM model support (Ollama, vLLM, LM Studio, LocalAI) and Office UI Settings - #30
Conversation
LZH-YS1998
left a comment
LZH-YS1998
left a comment
There was a problem hiding this comment.
Request changes
Thank you for working on native local/self-hosted LLM support. Supporting Ollama, vLLM, LM Studio, LocalAI, and other OpenAI-compatible servers is valuable, but I do not think this PR is safe to merge in its current form.
There are several blocking correctness issues that can either make the advertised feature non-functional or break existing cloud-provider configurations. In affected cases, native-agent work items will fail and remain in the in-progress column with a failed status, preventing dependent work from progressing.
1. The Office UI “Save & Apply” action does not apply anything
LLMModelSettingsModal.handleSave() only writes the settings to browser localStorage and invokes an optional onSave callback:
opc/plugins/office_ui/frontend_src/components/LLMModelSettingsModal.tsxhandleSave()around lines 93–111
However, App.tsx renders the modal without providing onSave:
<LLMModelSettingsModal
isOpen={showLLMModal}
onClose={() => setShowLLMModal(false)}
/>There is also no HTTP/WebSocket request that sends these settings to the backend, no update to llm_config.yaml, and no runtime LLMProvider reinitialization.
As a result:
- Clicking “Save & Apply Settings” reports success even though the backend configuration is unchanged.
- Selecting Ollama or another local model does not switch the model used by running work items.
- A user without valid backend credentials can believe that a keyless local model is active while native-agent work continues to fail.
- The README statement that the Office UI can configure the LLM is currently incorrect.
Additionally, storing API keys in localStorage is not appropriate. Any JavaScript running under the same origin can read them, and they remain in the browser profile indefinitely.
Required change
Please implement an actual backend configuration flow:
- Load the effective backend LLM configuration when opening the modal.
- Validate and persist changes through an appropriate backend endpoint.
- Store secrets on the backend, not in
localStorage. - Atomically update the configuration.
- Recreate/reconfigure the active
LLMProvider, or explicitly require and communicate a restart. - Return an error when validation or persistence fails instead of always showing success.
- Add an integration test proving that saving settings changes the model used by the next request.
2. Several advertised model prefixes are not supported by the pinned LiteLLM version
The project pins:
litellm==1.82.1I tested provider resolution against that exact installed version. The results were:
ollama/llama3.3 -> supported
ollama_chat/qwen2.5-coder -> supported
vllm/meta-llama-3.1-8b-instruct -> supported
hosted_vllm/model -> supported
lmstudio/deepseek-r1-distill-qwen-14b -> LLM Provider NOT provided
localai/starcoder2-15b -> LLM Provider NOT provided
llama-cpp/mistral-7b-instruct -> LLM Provider NOT provided
llamacpp/model -> LLM Provider NOT provided
tgi/model -> LLM Provider NOT provided
custom-model -> LLM Provider NOT provided
Unfortunately, several failing formats are used as UI defaults and documented as supported:
defaultModel: 'lmstudio/deepseek-r1-distill-qwen-14b'
defaultModel: 'localai/starcoder2-15b'
defaultModel: 'custom-model'These configurations fail before an HTTP request reaches the local server.
The new tests do not catch this because litellm.acompletion is mocked. They verify that arguments are assembled, but not that LiteLLM can resolve the advertised provider/model identifiers.
Required change
- Use model identifiers supported by the pinned LiteLLM version.
- For OpenAI-compatible servers, use the appropriate LiteLLM provider format together with
api_base. - Ensure the UI, documentation, and backend all use the same formats.
- Add non-network tests using LiteLLM’s real provider-resolution logic for every advertised provider.
- Do not claim support for LM Studio, LocalAI, llama.cpp, or TGI until their documented configuration passes provider resolution.
3. Local-provider environment variables can misroute cloud-model requests
LLMProvider.__init__() currently resolves api_base using:
self._api_base = config.api_base or (
os.environ.get("OLLAMA_API_BASE")
or os.environ.get("OLLAMA_HOST")
or os.environ.get("LOCAL_LLM_API_BASE")
or os.environ.get("OPENAI_API_BASE")
) or NoneThis is applied without checking the active model/provider.
I reproduced the following configuration:
OLLAMA_HOST=http://127.0.0.1:11434
default_model=anthropic/claude-sonnet-4-20250514
The resulting provider state was:
resolved_api_base=http://127.0.0.1:11434
has_credentials=True
This means that merely having Ollama configured in the environment can cause an Anthropic request to be sent to the Ollama endpoint.
_is_local_endpoint() has the same global behavior: the presence of any local-provider environment variable makes it return True, even if the model being evaluated is a cloud model. The request path may then inject:
api_key = "local"This can break existing cloud configurations on machines that also run a local model server.
Required change
- Resolve environment variables only for their matching provider.
- Do not treat the presence of a local-provider environment variable as evidence that an unrelated model is local.
- Derive
api_base, credentials, and local/keyless behavior from the model selected for the specific request. - Add regression tests for cloud models while
OLLAMA_HOST,VLLM_API_BASE, and similar variables are present.
4. Routed models inherit the default model’s endpoint
The provider calculates _api_base once from config.default_model, while _select_model(task_type) may later choose a different model from config.routing.
For example:
llm:
default_model: ollama/llama3.3
routing:
planning: anthropic/claude-sonnet-4-20250514The routed Anthropic request still inherits the Ollama endpoint. Because the inherited endpoint is local, the request can also receive the dummy "local" API key.
The reverse configuration is similarly unsafe: a cloud default with a routed local model does not reliably resolve the local model’s endpoint.
Required change
Resolve endpoint and authentication settings per selected model/request, not once from default_model. Please add tests for both directions:
- Local default → cloud routed model
- Cloud default → local routed model
5. Switching from a local provider to a cloud provider preserves the local URL
The UI only updates apiBase when the selected preset has a non-empty default:
if (preset.defaultBase) setApiBase(preset.defaultBase)Cloud presets use an empty defaultBase. Therefore:
- Select Ollama.
api_basebecomeshttp://localhost:11434.- Select OpenAI or Anthropic.
- The Ollama URL remains in the form and is saved with the cloud model.
This is currently masked by the missing backend integration. Once the save flow is implemented, it will misroute cloud requests.
Required change
Always set the API base when changing provider, including clearing it for providers that use their standard endpoint.
6. Local capability reporting is overly optimistic
get_capabilities() reports tool calling, streaming tool calls, and streaming support as enabled for all models. Local models and local inference servers do not necessarily support those features.
OpenOPC’s work-item runtime depends heavily on reliable tool calling. A model that only supports text generation may appear successfully configured but fail to execute the requested work, loop without useful actions, or ultimately return a failed result.
Required change
At minimum:
- Document the tool-calling requirement.
- Add a connection/model validation step.
- Avoid presenting an endpoint as fully operational based only on its hostname or model prefix.
- Ideally verify basic completion and tool-call support before applying the configuration.
7. The PR contains unrelated and incomplete Shadow Adapter changes
The PR also changes:
opc/cli/app.pyopc/layer3_agent/adapters/registry.pyopc/plugins/shadow_adapter/__init__.py
These changes are unrelated to local LLM support. The plugin imports a top-level shadow_adapter package that is neither included nor declared as a dependency.
The imports are partially protected by ImportError, so the normal missing-package case may be ignored, but the feature is incomplete and expands the regression surface of this PR. An incompatible or partially installed package could also raise a non-ImportError exception during startup.
Required change
Please remove the Shadow Adapter changes from this PR and submit them separately with their implementation, dependency declaration, tests, and documentation.
Work-item impact
A provider-resolution or endpoint-routing error follows this path:
- The LLM stream fails.
- The runtime performs bounded provider-error retries.
- The runtime returns
TaskStatus.FAILED. - The engine may retry the task once according to its retry policy.
- The final task remains failed.
- Office UI keeps the task in the
in-progresscolumn withstatus=failed. - Downstream work items that depend on its output cannot progress normally.
This normally does not leave the task permanently marked as running, but it does block the work item until a user corrects the configuration and retries it.
Verification performed
I reviewed the net diff from the current main base to PR head 183b51adf6921f6ad39ad072a3dbb0a5d43bd1c4 and ran the following checks against the PR snapshot:
- PR-declared backend test selection: 115 passed
- Native runtime, company collaboration, and Office UI status tests: 223 passed
- TypeScript type check: passed
git diff --check: passed
These results show that the existing tests remain green, but they do not cover the blocking scenarios above. In particular, there is no real LiteLLM provider-resolution test, no UI-to-backend configuration test, and no cross-provider environment/routing test.
Decision
Request changes.
The feature is worth pursuing, and a limited Ollama/vLLM implementation could be a good first step. However, the current PR should not be merged until:
- The UI settings actually reach and update the backend.
- API keys are removed from browser
localStorage. - Every advertised model format works with the pinned LiteLLM version.
- Endpoint and credential resolution is scoped per provider and selected model.
- Mixed routing is tested.
- Local-to-cloud provider switching clears stale endpoint values.
- End-to-end work-item failure/recovery tests are added.
- The unrelated Shadow Adapter changes are removed or split into a separate PR.
|
Hi @LZH-YS1998, Thank you for the thorough and constructive code review. I've addressed all 7 blocking issues in our latest commits and merged the latest 1. Office UI "Save & Apply" Backend Configuration Flow
2. LiteLLM Provider Resolution for Advertised Models
3. Provider-Scoped Environment Variables
4. Dynamic Per-Model Endpoint & Routing Resolution
5. Local-to-Cloud Provider Preset Switching
6. Local Capability Guidance
7. Clean PR Scope
Verification Run
Please let me know if any further adjustments are needed! |
LZH-YS1998
left a comment
LZH-YS1998
left a comment
There was a problem hiding this comment.
Thank you for addressing the first review. I re-reviewed and executed the current head (3bf6c767) rather than only reading the new tests. The provider-format cleanup and removal of the unrelated shadow adapter are good, but the backend/runtime path still has release-blocking correctness and credential-isolation problems.
-
The Office UI reads and writes the wrong configuration directory. In
server.py:225-228,253-266and the equivalent WS handler,opc_homeis passed toOPCConfig.load/save, while the application starts fromopc_home / "config". Also,create_app()storesengine, notapp["opc_home"]orapp["config"], so the fallback path is always used. I reproduced a POST returning HTTP 200 while the canonicalopc_home/config/llm_config.yamlremained on the old Anthropic model/key; the handler instead createdsystem_config.yaml,llm_config.yaml,agent_config.yaml, andchannel_config.yamldirectly underopc_home. This means the UI reports success but the setting is lost on restart, and a nominal LLM-only update writes unrelated config files. Please use the canonical config directory, persist only the intended LLM config under the existing config lock/service boundary, and add a real HTTP/WS handler integration test that starts withopc_home/config/llm_config.yamland verifies that exact file. -
“Hot apply” does not reconfigure the running engine.
server.py:268-271only replacesengine.llm. After initializing a realOPCEngineand applying the same replacement,history_compactor,communication,approval_engine,secretary,company_runtime_spec_builder,company_recruiter,company_executor, andtask_routerall still referenced the oldLLMProvider. Company work items can therefore mix old and new models/endpoints in one run. Project delegate engines and already-running work also need an explicit policy. Please provide one engine-level reconfiguration operation that consistently updates all consumers (with safe in-flight semantics), or make the UI honestly require a restart. The current test namedtest_backend_config_persistence_and_provider_reinitializationnever calls a handler or initializes an engine, so it cannot validate this claim. -
Mixed local/cloud routing is still misrouted.
resolve_api_base()returns the single explicitconfig.api_basebefore inspecting the selected routed model (provider.py:330-375), andconfig.is_localis likewise applied globally (provider.py:377-388,711-716). With the configuration produced by the UI—Ollama default,api_base=http://localhost:11434,is_local=true, and an Anthropic planning route—I captured the actuallitellm.acompletionkwargs:
model=anthropic/claude-sonnet-4-20250514
api_base=http://localhost:11434
api_key=local
That cloud planning/approval call goes to Ollama and can block/fail the work item. The added routing test avoids both explicit UI fields and checks only resolve_api_base, not the actual request kwargs. Endpoint, key, key-env, and local/keyless state must be scoped to the selected model/provider. Please test actual acompletion kwargs in both local→cloud and cloud→local directions, including explicit bases, is_local, configured keys, and api_key_env.
-
Stale cloud credentials can be sent to a local server. The update API treats blank and
"***"only as “keep existing key”; there is no clear operation, andapi_key_envis never cleared or scoped. Switching from a cloud provider to Ollama with a blank key retainedcloud-secretin my reproduction, and the new runtime provider then sent that secret tohttp://localhost:11434. Please define explicit keep/set/clear key semantics and ensure credentials for one provider are never forwarded to another provider or a keyless local endpoint. -
The configuration flow is duplicated and insufficiently validated. The React UI uses REST, but a second near-identical GET/update implementation was added to WS plus unused client callbacks. Neither path shares validation or consistently uses
_config_lock; model/provider/base consistency and URL/context-window ranges are not validated. Please choose one path or put both thin transports over one tested service so behavior cannot drift and concurrent full-config saves cannot overwrite unrelated changes. -
Please rebase onto current
mainand rebuild the committed frontend bundle. The current PR isDIRTY; an isolated merge has a content conflict infrontend_dist/index.html. Currentmainalso adds workspace-authority trust/fingerprint enforcement, so an authorized UI config mutation must integrate with that mechanism rather than bypassing it via the wrong directory.
Verification performed: the PR's 10 new Python tests pass, frontend typecheck/build and contract tests pass, and 69 related provider/server/workspace-trust tests pass after allowing the two loopback-only trust tests outside the socket-restricted sandbox. Those green tests do not cover the failures above, and the current PR head has no GitHub check runs.
…y, engine hot-apply, and credential isolation
|
Hi @LZH-YS1998, Thanks for the detailed second review :) I went through all of the remaining points and addressed them in the latest commits. I’ve also merged/rebased the changes onto the latest The main changes are: 1. Canonical Config Directory & Atomic File Persistence
2. Engine-Wide Reconfiguration ("Hot Apply")
3. Model-Scoped Endpoint & Credential Isolation (Mixed Routing Fix)
4. Explicit Key Semantics & Credential Leak Guard
5. Transport Deduplication (
|
2 participants
Overview
This PR adds support for using local/self-hosted LLM providers with OpenOPC without requiring a remote API key.
Supported local providers include Ollama, vLLM, LM Studio, LocalAI, Llama.cpp, and TGI. It also adds an LLM settings modal to the Office UI so the provider, model, API base, and API key can be configured directly from the UI.
The existing cloud providers are still supported.
Changes
1. Local LLM provider support
Updated
opc/llm/provider.pyto handle local/self-hosted providers.has_credentials()now considers supported local providers valid without requiring an API key.Added automatic
api_basedefaults for local providers when no endpoint is configured:ollama/→http://localhost:11434vllm/→http://localhost:8000/v1localai/→http://localhost:8080/v1lmstudio/→http://localhost:1234/v1Added the
api_key = "local"fallback for keyless OpenAI-compatible local endpoints. This prevents the underlying client from failing because of a missing API key/header.Updated
get_capabilities()to expose local provider information, includingis_localand the provider family.2. Office UI LLM settings
Added
LLMModelSettingsModal.tsxand integrated it intoApp.tsx.The Office UI now has an LLM settings button in the top navigation header.
The provider selector includes:
The settings UI also handles local and cloud providers differently:
Local provider endpoints are pre-filled with their default
api_base.API keys are optional for local providers.
API keys remain required for providers that need them.
A local-mode indicator is shown when a keyless local provider is selected.
Added quick-select model options for commonly used local models, including:
ollama/llama3.3vllm/meta-llama-3.1-8b-instructlmstudio/deepseek-r1-distill-qwen-14b3. LLM configuration schema
Updated
opc/core/config.pyand extendedLLMConfigwith:This allows the configuration to keep track of the selected provider and whether it is running in local mode.
4. Tests
Added:
The new tests cover the local provider handling and keyless configuration behavior.
Also verified the existing LLM and CLI-related tests.
5. Documentation
Added:
and updated:
to document the local model setup and supported providers.
Verification
Ran the backend test suite with:
Result:
115/115 tests passed
Ran the TypeScript check in the Office UI frontend:
Result:
0 errors
Also rebuilt the Office UI production bundle:
The Vite production build completed successfully and updated:
Upstream sync
Before pushing the final changes, I fetched and merged the latest
upstream/mainchanges.The merge introduced a conflict in the frontend bundle, which was resolved and rebuilt.
The merged changes were committed and pushed to
origin/main.I also removed the
openopc-shadow-adapterdirectory reference from the core git index and pushed that fix.The branch is now up to date with
upstream/mainand has no remaining merge conflicts.