feat: allow model registration without provider API keys (#5014)

# What does this PR do?
Related to #5013.

Add a `model_validation` configuration option for models that controls
whether to validate model availability during registration. This enables
multi-tenant deployments where llama-stack can start without API keys
and accept per-request authentication via the
`X-LlamaStack-Provider-Data` header.

## Key Changes

1. **Model-level `model_validation` field** (`ModelInput`,
`RegisterModelRequest`):
- `model_validation: false` (default) - Skip validation during
registration, defer to runtime
- `model_validation: true` - Validate model exists during registration
(requires API key)
- Models are preserved during provider refresh when validation is
disabled

2. **Updated routing table logic**:
- Models registered via API are preserved during refresh (not removed)
   - Debug logging added for skipped refreshes

## Test Plan

### Unit Tests
Added 11 unit tests to `test_openai_mixin.py` covering:
- Default behavior (no validation)
- Model-level validation enabled/disabled
- Interaction with `refresh_models` setting
- Model registration success/failure scenarios

### Integration Test

Started a llama-stack server with 4 providers to test different
scenarios:

<details>
<summary>Test Configuration</summary>

```yaml
providers:
  inference:
    # Scenario 1: No API key, no refresh (Gemini)
    - provider_id: gemini
      provider_type: remote::gemini
      config: {}

    # Scenario 2: No API key, no refresh (Anthropic)
    - provider_id: anthropic
      provider_type: remote::anthropic
      config: {}

    # Scenario 3: No API key, no refresh (OpenAI)
    - provider_id: openai-skip
      provider_type: remote::openai
      config: {}

    # Scenario 4: Real API key, refresh enabled (OpenAI)
    - provider_id: openai-real
      provider_type: remote::openai
      config:
        refresh_models: true
        api_key: sk-proj-...

registered_resources:
  models:
    # Default behavior: no validation
    - provider_id: gemini
      model_id: models/gemini-2.5-flash-lite
      model_type: llm

    # Default behavior: no validation
    - provider_id: anthropic
      model_id: claude-haiku-4-5-20251001
      model_type: llm

    # Default behavior: no validation
    - provider_id: openai-skip
      model_id: gpt-4o-mini
      model_type: llm

    # Explicit validation during registration
    - provider_id: openai-real
      model_id: gpt-4o-mini
      model_type: llm
      model_validation: true
```

</details>

### Results

✅ **All tests passed**

1. **Server startup**: All models registered successfully without
requiring API keys
2. **Models API (`/v1/models`)**: All 4 test models appear in the
response
3. **Inference without API key**: Providers 1-3 fail at runtime
(expected), provider 4 succeeds with configured key
4. **Inference with per-request API key** (via
`X-LlamaStack-Provider-Data` header): All providers succeed

<details>
<summary>Detailed Test Results</summary>

### 1. Server Startup
```bash
$ llama stack run llama_stack/configs/test-skip-model-availability.yaml --port 8322
```

✅ Server started successfully. Models with no API keys registered
without errors (validation deferred to runtime).

### 2. Verify Models Registered
```bash
$ curl -s http://localhost:8322/v1/models | jq -r '.data[].id' | grep -E '(gemini/|anthropic/|openai-skip/|openai-real/)' | head -10
```

✅ Output:
```
gemini/models/gemini-2.5-flash-lite
anthropic/claude-haiku-4-5-20251001
openai-skip/gpt-4o-mini
openai-real/gpt-4o-mini
```

(Note: openai-real also auto-discovered 116+ additional models via
`refresh_models: true`)

### 3. Inference Without Provider Data Header

#### Gemini (no API key)
```bash
$ curl -s http://localhost:8322/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gemini/models/gemini-2.5-flash-lite", "messages": [{"role": "user", "content": "Say hello"}], "max_tokens": 50}'
```

✅ Expected failure:
```json
{"error": {"message": "API key not valid. Please pass a valid API key."}}
```

#### Anthropic (no API key)
```bash
$ curl -s http://localhost:8322/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "anthropic/claude-haiku-4-5-20251001", "messages": [{"role": "user", "content": "Say hello"}], "max_tokens": 50}'
```

✅ Expected failure:
```json
{"error": {"message": "Invalid Anthropic API Key"}}
```

#### OpenAI-skip (no API key)
```bash
$ curl -s http://localhost:8322/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "openai-skip/gpt-4o-mini", "messages": [{"role": "user", "content": "Say hello"}], "max_tokens": 50}'
```

✅ Expected failure:
```json
{"error": {"message": "API key not provided. Please provide a valid API key in the provider data header..."}}
```

#### OpenAI-real (configured API key)
```bash
$ curl -s http://localhost:8322/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "openai-real/gpt-4o-mini", "messages": [{"role": "user", "content": "Say hello"}], "max_tokens": 50}'
```

✅ Success:
```json
{
  "choices": [{
    "message": {"content": "Hello! How can I assist you today?", "role": "assistant"},
    "finish_reason": "stop"
  }],
  "usage": {"completion_tokens": 9, "prompt_tokens": 9, "total_tokens": 18}
}
```

### 4. Inference With Per-Request API Keys

#### Anthropic with provider data header
```bash
$ curl -s http://localhost:8322/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H 'X-LlamaStack-Provider-Data: {"anthropic_api_key": "sk-ant-api03-..."}' \
  -d '{"model": "anthropic/claude-haiku-4-5-20251001", "messages": [{"role": "user", "content": "Say hello"}], "max_tokens": 50}'
```

✅ Success:
```json
{
  "choices": [{
    "message": {"content": "Hello! 👋 How can I help you today?", "role": "assistant"},
    "finish_reason": "stop"
  }],
  "usage": {"completion_tokens": 21, "prompt_tokens": 10, "total_tokens": 31}
}
```

</details>

## Summary

This PR enables multi-tenant deployments where:
- **Models can be registered without API keys** (validation deferred to
runtime)
- **Per-request authentication works** via `X-LlamaStack-Provider-Data`
header
- **Explicit validation is opt-in** via `model_validation: true` on
specific models
- **Provider refresh is independent** and controlled by existing
`refresh_models` setting

Author: NickGagan

client-sdks/stainless/openapi.yml

@@ -7032,6 +7032,11 @@ components:
         model_type:
           $ref: '#/components/schemas/ModelType'
           default: llm
+        model_validation:
+          anyOf:
+          - type: boolean
+          - type: 'null'
+          description: Enable model availability check during registration. When false (default), validation is deferred to runtime and model is preserved during provider refresh.
       type: object
       required:
       - identifier
@@ -12022,6 +12027,11 @@ components:
           - type: 'null'
           description: The type of model to register.
           title: ModelType
+        model_validation:
+          anyOf:
+          - type: boolean
+          - type: 'null'
+          description: Enable model availability check during registration. When false (default), validation is deferred to runtime and model is preserved during provider refresh.
       type: object
       required:
       - model_id

docs/static/deprecated-llama-stack-spec.yaml

@@ -3576,6 +3576,11 @@ components:
         model_type:
           $ref: '#/components/schemas/ModelType'
           default: llm
+        model_validation:
+          anyOf:
+          - type: boolean
+          - type: 'null'
+          description: Enable model availability check during registration. When false (default), validation is deferred to runtime and model is preserved during provider refresh.
       type: object
       required:
       - identifier
@@ -8566,6 +8571,11 @@ components:
           - type: 'null'
           description: The type of model to register.
           title: ModelType
+        model_validation:
+          anyOf:
+          - type: boolean
+          - type: 'null'
+          description: Enable model availability check during registration. When false (default), validation is deferred to runtime and model is preserved during provider refresh.
       type: object
       required:
       - model_id

docs/static/experimental-llama-stack-spec.yaml

@@ -3685,6 +3685,11 @@ components:
         model_type:
           $ref: '#/components/schemas/ModelType'
           default: llm
+        model_validation:
+          anyOf:
+          - type: boolean
+          - type: 'null'
+          description: Enable model availability check during registration. When false (default), validation is deferred to runtime and model is preserved during provider refresh.
       type: object
       required:
       - identifier
@@ -8646,6 +8651,11 @@ components:
           - type: 'null'
           description: The type of model to register.
           title: ModelType
+        model_validation:
+          anyOf:
+          - type: boolean
+          - type: 'null'
+          description: Enable model availability check during registration. When false (default), validation is deferred to runtime and model is preserved during provider refresh.
       type: object
       required:
       - model_id

docs/static/llama-stack-spec.yaml

@@ -5453,6 +5453,11 @@ components:
         model_type:
           $ref: '#/components/schemas/ModelType'
           default: llm
+        model_validation:
+          anyOf:
+          - type: boolean
+          - type: 'null'
+          description: Enable model availability check during registration. When false (default), validation is deferred to runtime and model is preserved during provider refresh.
       type: object
       required:
       - identifier
@@ -10425,6 +10430,11 @@ components:
           - type: 'null'
           description: The type of model to register.
           title: ModelType
+        model_validation:
+          anyOf:
+          - type: boolean
+          - type: 'null'
+          description: Enable model availability check during registration. When false (default), validation is deferred to runtime and model is preserved during provider refresh.
       type: object
       required:
       - model_id

docs/static/stainless-llama-stack-spec.yaml

@@ -7032,6 +7032,11 @@ components:
         model_type:
           $ref: '#/components/schemas/ModelType'
           default: llm
+        model_validation:
+          anyOf:
+          - type: boolean
+          - type: 'null'
+          description: Enable model availability check during registration. When false (default), validation is deferred to runtime and model is preserved during provider refresh.
       type: object
       required:
       - identifier
@@ -12022,6 +12027,11 @@ components:
           - type: 'null'
           description: The type of model to register.
           title: ModelType
+        model_validation:
+          anyOf:
+          - type: boolean
+          - type: 'null'
+          description: Enable model availability check during registration. When false (default), validation is deferred to runtime and model is preserved during provider refresh.
       type: object
       required:
       - model_id

src/llama_stack/core/routing_tables/models.py

@@ -46,7 +46,12 @@ async def refresh(self) -> None:
             try:
                 models = await provider.list_models()
             except Exception as e:
-                logger.warning(f"Model refresh failed for provider {provider_id}: {e}")
+                if provider_id not in self.listed_providers:
+                    # Mark provider as listed to prevent repeated refresh attempts
+                    self.listed_providers.add(provider_id)
+                    logger.warning(f"Model refresh skipped for provider {provider_id}")
+                else:
+                    logger.warning(f"Model refresh failed for provider {provider_id}: {e}")
                 continue
 
             self.listed_providers.add(provider_id)
@@ -210,14 +215,16 @@ async def register_model(
         provider_id: str | None = None,
         metadata: dict[str, Any] | None = None,
         model_type: ModelType | None = None,
+        model_validation: bool | None = None,
     ) -> Model:
         # Support both the public Models API (RegisterModelRequest) and legacy parameter-based interface
         if isinstance(request, RegisterModelRequest):
             model_id = request.model_id
             provider_model_id = request.provider_model_id
             provider_id = request.provider_id
-            metadata = request.metadata
+            metadata = request.metadata or {}
             model_type = request.model_type
+            model_validation = request.model_validation
         elif isinstance(request, str):
             # Legacy positional argument: register_model("model-id", ...)
             model_id = request
@@ -248,6 +255,7 @@ async def register_model(
             provider_id=provider_id,
             metadata=metadata,
             model_type=model_type,
+            model_validation=model_validation,
             source=RegistryEntrySource.via_register_api,
         )
         registered_model = await self.register_object(model)

src/llama_stack/providers/utils/inference/openai_mixin.py

@@ -480,6 +480,13 @@ async def openai_embeddings(
     ##
 
     async def register_model(self, model: Model) -> Model:
+        # Check if we should validate model availability (defaults to False)
+        should_validate = bool(model.model_validation)
+
+        if not should_validate:
+            logger.debug(f"Skipping model availability check for {model.provider_model_id} (model_validation=false)")
+            return model
+
         if not await self.check_model_availability(model.provider_model_id):
             raise ValueError(f"Model {model.provider_model_id} is not available from provider {self.__provider_id__}")  # type: ignore[attr-defined]
         return model

src/llama_stack_api/models/models.py

@@ -66,6 +66,10 @@ def provider_model_id(self) -> str:
     model_config = ConfigDict(protected_namespaces=())
 
     model_type: ModelType = Field(default=ModelType.llm)
+    model_validation: bool | None = Field(
+        default=None,
+        description="Enable model availability check during registration. When false (default), validation is deferred to runtime and model is preserved during provider refresh.",
+    )
 
     @field_validator("provider_resource_id")
     @classmethod
@@ -134,6 +138,10 @@ class RegisterModelRequest(BaseModel):
     provider_id: str | None = Field(default=None, description="The identifier of the provider.")
     metadata: dict[str, Any] | None = Field(default=None, description="Any additional metadata for this model.")
     model_type: ModelType | None = Field(default=None, description="The type of model to register.")
+    model_validation: bool | None = Field(
+        default=None,
+        description="Enable model availability check during registration. When false (default), validation is deferred to runtime and model is preserved during provider refresh.",
+    )
 
 
 @json_schema_type

tests/unit/providers/utils/inference/test_openai_mixin.py

@@ -515,13 +515,18 @@ async def test_check_model_availability_with_allowed_models(
 class TestOpenAIMixinModelRegistration:
     """Test cases for model registration functionality"""
 
-    async def test_register_model_success(self, mixin, mock_client_with_models, mock_client_context):
+    async def test_register_model_success(self, mock_client_with_models, mock_client_context):
         """Test successful model registration when model is available"""
+        config = RemoteInferenceProviderConfig()
+        mixin = OpenAIMixinImpl(config=config)
+
+        # Enable validation for this model
         model = Model(
             provider_id="test-provider",
             provider_resource_id="some-mock-model-id",
             identifier="test-model",
             model_type=ModelType.llm,
+            model_validation=True,
         )
 
         with mock_client_context(mixin, mock_client_with_models):
@@ -534,13 +539,18 @@ async def test_register_model_success(self, mixin, mock_client_with_models, mock
             assert result.model_type == ModelType.llm
             mock_client_with_models.models.list.assert_called_once()
 
-    async def test_register_model_not_available(self, mixin, mock_client_with_models, mock_client_context):
+    async def test_register_model_not_available(self, mock_client_with_models, mock_client_context):
         """Test model registration failure when model is not available from provider"""
+        config = RemoteInferenceProviderConfig()
+        mixin = OpenAIMixinImpl(config=config)
+
+        # Enable validation for this model
         model = Model(
             provider_id="test-provider",
             provider_resource_id="non-existent-model",
             identifier="test-model",
             model_type=ModelType.llm,
+            model_validation=True,
         )
 
         with mock_client_context(mixin, mock_client_with_models):
@@ -550,24 +560,27 @@ async def test_register_model_not_available(self, mixin, mock_client_with_models
                 await mixin.register_model(model)
             mock_client_with_models.models.list.assert_called_once()
 
-    async def test_register_model_with_allowed_models_filter(self, mixin, mock_client_with_models, mock_client_context):
+    async def test_register_model_with_allowed_models_filter(self, mock_client_with_models, mock_client_context):
         """Test model registration with allowed_models filtering"""
-        mixin.config.allowed_models = ["some-mock-model-id"]
+        config = RemoteInferenceProviderConfig(allowed_models=["some-mock-model-id"])
+        mixin = OpenAIMixinImpl(config=config)
 
-        # Test with allowed model
+        # Test with allowed model (with validation enabled)
         allowed_model = Model(
             provider_id="test-provider",
             provider_resource_id="some-mock-model-id",
             identifier="allowed-model",
             model_type=ModelType.llm,
+            model_validation=True,
         )
 
-        # Test with disallowed model
+        # Test with disallowed model (with validation enabled)
         disallowed_model = Model(
             provider_id="test-provider",
             provider_resource_id="final-mock-model-id",
             identifier="disallowed-model",
             model_type=ModelType.llm,
+            model_validation=True,
         )
 
         with mock_client_context(mixin, mock_client_with_models):
@@ -616,25 +629,99 @@ async def test_should_refresh_models(self, mixin):
         result = await mixin.should_refresh_models()
         assert result is False
 
+        # With refresh_models=True, should return True
         config_with_refresh = RemoteInferenceProviderConfig(refresh_models=True)
         mixin_with_refresh = OpenAIMixinImpl(config=config_with_refresh)
         result_with_refresh = await mixin_with_refresh.should_refresh_models()
         assert result_with_refresh is True
 
-    async def test_register_model_error_propagation(self, mixin, mock_client_with_exception, mock_client_context):
+    async def test_register_model_error_propagation(self, mock_client_with_exception, mock_client_context):
         """Test that errors from provider API are properly propagated during registration"""
+        config = RemoteInferenceProviderConfig()
+        mixin = OpenAIMixinImpl(config=config)
+
+        # Enable validation for this model
         model = Model(
             provider_id="test-provider",
             provider_resource_id="some-model",
             identifier="test-model",
             model_type=ModelType.llm,
+            model_validation=True,
         )
 
         with mock_client_context(mixin, mock_client_with_exception):
             # The exception from the API should be propagated
             with pytest.raises(Exception, match="API Error"):
                 await mixin.register_model(model)
 
+    async def test_register_model_default_behavior_no_validation(self, mock_client_with_models, mock_client_context):
+        """Test model registration with default behavior (no validation)"""
+        # Default behavior - no validation
+        config = RemoteInferenceProviderConfig()
+        mixin = OpenAIMixinImpl(config=config)
+
+        model = Model(
+            provider_id="test-provider",
+            provider_resource_id="non-existent-model",
+            identifier="test-model",
+            model_type=ModelType.llm,
+        )
+
+        with mock_client_context(mixin, mock_client_with_models):
+            # Should succeed without checking model availability (default behavior)
+            result = await mixin.register_model(model)
+
+            assert result == model
+            # Verify that models.list() was NOT called
+            mock_client_with_models.models.list.assert_not_called()
+
+    async def test_register_model_with_validation_enabled(self, mock_client_with_models, mock_client_context):
+        """Test that model-level model_validation=True enables validation"""
+        # Default config (no provider-level validation setting)
+        config = RemoteInferenceProviderConfig()
+        mixin = OpenAIMixinImpl(config=config)
+
+        # Model explicitly enables validation
+        model = Model(
+            provider_id="test-provider",
+            provider_resource_id="non-existent-model",
+            identifier="test-model",
+            model_type=ModelType.llm,
+            model_validation=True,
+        )
+
+        with mock_client_context(mixin, mock_client_with_models):
+            # Should fail because model-level validation is enabled
+            with pytest.raises(ValueError, match="Model non-existent-model is not available"):
+                await mixin.register_model(model)
+            # Verify that models.list() WAS called (validation happened)
+            mock_client_with_models.models.list.assert_called_once()
+
+    async def test_register_model_with_validation_explicitly_disabled(
+        self, mock_client_with_models, mock_client_context
+    ):
+        """Test that model-level model_validation=False explicitly disables validation"""
+        # Default config
+        config = RemoteInferenceProviderConfig()
+        mixin = OpenAIMixinImpl(config=config)
+
+        # Model explicitly disables validation (though this is the default anyway)
+        model = Model(
+            provider_id="test-provider",
+            provider_resource_id="non-existent-model",
+            identifier="test-model",
+            model_type=ModelType.llm,
+            model_validation=False,
+        )
+
+        with mock_client_context(mixin, mock_client_with_models):
+            # Should succeed because validation is disabled
+            result = await mixin.register_model(model)
+
+            assert result == model
+            # Verify that models.list() was NOT called
+            mock_client_with_models.models.list.assert_not_called()
+
 
 class ProviderDataValidator(BaseModel):
     """Validator for provider data in tests"""