fix: make the provider connection test resilient to model retirement (#1035)

* fix: make the provider connection test resilient to model retirement

Two independent hardenings against the #970 class of breakage (a hard-coded
Gemini test model getting shut down by Google, which made testing a valid
key fail with a 404):

- Use Google's floating alias gemini-flash-latest for the Google/Vertex
  test model instead of a dated id, so a retirement repoints it for us.
- Reframe the provider connection test around what an error actually
  proves: only a rejected key (401), missing permissions (403), or an
  unreachable endpoint are failures. Anything the provider returns after
  authenticating - a rate limit, or a missing/retired/unsupported model -
  still proves the credentials work, so it reports success. Previously this
  relied on matching the literal phrase 'not found' + 'model', which a
  differently-worded retirement/deprecation error slipped past.

Unifies the auth/network/rate-limit classification (previously duplicated
and divergent between connection_tester and credentials_service) into
shared helpers. The individual-model test keeps model-not-found as a
failure, since there a specific registered model really is broken.

Adds classification tests with realistic provider error strings.

* docs: add CHANGELOG entry for connection-test resilience fix

Author: lfnovo

CHANGELOG.md

@@ -32,6 +32,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `docker-compose.yml` now sources the SurrealDB credentials from `SURREAL_USER` / `SURREAL_PASSWORD` (applied to both the database server and the app), defaulting to `root:root` so the zero-config quick start is unchanged. Set them in a `.env` file to use your own credentials before exposing the instance; `.env.example` and the compose file note this (#946)
 
 ### Fixed
+- Testing a valid Google/Vertex credential no longer fails after Google retires a Gemini model. The connection test used a hard-coded model id that Google shuts down on a schedule (`gemini-2.0-flash`), so a valid key surfaced as a broken connection (#970). The Google/Vertex test now uses Google's floating `gemini-flash-latest` alias, and the provider connection test was reframed so only a rejected key, missing permissions, or an unreachable endpoint count as failures — a missing/retired/rate-limited model still reports the credentials as valid
 - API startup no longer crashes when SurrealDB isn't ready yet (e.g. docker-compose race on host reboot: `Temporary failure in name resolution`). The lifespan now polls a lightweight readiness probe with bounded exponential backoff (~50s budget, 5s per-probe timeout) before running migrations; migration errors themselves still fail fast (#708)
 - Markdown typography styles (`prose` classes) are active again: the Tailwind v4 migration left the old `tailwind.config.ts` (which loaded `@tailwindcss/typography`) silently ignored, so rendered markdown lost its typographic styling. The plugin and class-based dark mode are now configured in `globals.css`, and markdown rendering is centralized in a shared `MarkdownRenderer` component (#783)
 - Podcast generation no longer truncates on dense, long-form content (`LengthFinishReasonError` / `OUTPUT_PARSING_FAILURE`): episode profiles now support an optional `max_tokens` that is passed through to podcast_creator's outline/transcript generation, overriding its defaults — settable via the episode profile API (UI follow-up in #991) (#639)

api/credentials_service.py

@@ -269,6 +269,7 @@ async def test_credential(credential_id: str) -> dict:
             _test_azure_connection,
             _test_ollama_connection,
             _test_openai_compatible_connection,
+            classify_provider_test_error,
         )
 
         provider = cred.provider.lower()
@@ -360,18 +361,10 @@ async def test_credential(credential_id: str) -> dict:
             }
 
         error_msg = str(e)
-        if "401" in error_msg or "unauthorized" in error_msg.lower():
-            return {"provider": provider, "success": False, "message": "Invalid API key"}
-        elif "403" in error_msg or "forbidden" in error_msg.lower():
-            return {"provider": provider, "success": False, "message": "API key lacks required permissions"}
-        elif "rate" in error_msg.lower() and "limit" in error_msg.lower():
-            return {"provider": provider, "success": True, "message": "Rate limited - but connection works"}
-        elif "not found" in error_msg.lower() and "model" in error_msg.lower():
-            return {"provider": provider, "success": True, "message": "API key valid (test model not available)"}
-        else:
+        success, message = classify_provider_test_error(error_msg)
+        if not success:
             logger.debug(f"Test connection error for credential {credential_id}: {e}")
-            truncated = error_msg[:100] + "..." if len(error_msg) > 100 else error_msg
-            return {"provider": provider, "success": False, "message": f"Error: {truncated}"}
+        return {"provider": provider, "success": success, "message": message}
 
 
 async def discover_with_config(provider: str, config: dict) -> List[dict]:

open_notebook/ai/connection_tester.py

@@ -49,10 +49,18 @@ def _is_vertex_credentials_file_error(exc: Exception) -> bool:
 
 # Test models for each provider - uses minimal/cheapest models for testing
 # Format: (model_name, model_type)
+#
+# Prefer a provider-maintained floating alias where one exists, so a model
+# retirement doesn't silently break the connection test (see #970: Google
+# hard-shuts-down Gemini model ids on a schedule). `gemini-flash-latest`
+# is Google's alias for the current stable Flash model and moves forward on
+# its own. The provider test also no longer treats a model-level failure as
+# a connection failure (see `_connection_failure_reason`), so even if an
+# alias ever breaks, the test still reports the credentials correctly.
 TEST_MODELS = {
     "openai": ("gpt-3.5-turbo", "language"),
     "anthropic": ("claude-3-haiku-20240307", "language"),
-    "google": ("gemini-3.5-flash", "language"),
+    "google": ("gemini-flash-latest", "language"),
     "groq": ("llama-3.1-8b-instant", "language"),
     "mistral": ("mistral-small-latest", "language"),
     "deepseek": ("deepseek-chat", "language"),
@@ -63,7 +71,7 @@ def _is_vertex_credentials_file_error(exc: Exception) -> bool:
     "deepgram": ("aura-2-thalia-en", "text_to_speech"),
     "ollama": (None, "language"),  # Dynamic - will use first available model
     # Complex providers with additional configuration
-    "vertex": ("gemini-3.5-flash", "language"),  # Uses Google Vertex AI
+    "vertex": ("gemini-flash-latest", "language"),  # Uses Google Vertex AI
     "azure": ("gpt-35-turbo", "language"),  # Azure OpenAI deployment name
     "openai_compatible": (None, "language"),  # Dynamic - will use first available model
     "dashscope": ("qwen-plus", "language"),
@@ -281,26 +289,114 @@ def _get_test_audio() -> io.BytesIO:
         return _generate_test_wav()
 
 
-def _normalize_error_message(error_msg: str) -> Tuple[bool, str]:
-    """Normalize common error patterns into user-friendly messages."""
+def _connection_failure_reason(error_msg: str) -> Optional[str]:
+    """Classify whether an error means the provider is genuinely unreachable
+    or the credentials are rejected.
+
+    Returns a user-facing failure message for the only errors that actually
+    disprove a working provider connection — bad key (401), insufficient
+    permissions (403), and network/timeout failures. Returns None for
+    anything the provider itself returned *after* authenticating (a missing
+    or retired model, an unsupported request, a rate limit): reaching the
+    model layer at all proves the credentials and endpoint work, so those
+    are not connection failures. This is what keeps a retired test model
+    (see #970) from being misreported as a broken provider connection.
+    """
     lower = error_msg.lower()
 
     if "401" in error_msg or "unauthorized" in lower:
-        return False, "Invalid API key"
-    elif "403" in error_msg or "forbidden" in lower:
-        return False, "API key lacks required permissions"
-    elif "rate" in lower and "limit" in lower:
+        return "Invalid API key"
+    if "403" in error_msg or "forbidden" in lower:
+        return "API key lacks required permissions"
+    if "timeout" in lower or "timed out" in lower:
+        return "Connection timed out - check network/endpoint"
+    if (
+        "connection" in lower
+        or "network" in lower
+        or "getaddrinfo" in lower
+        or "name resolution" in lower
+        or "failed to establish" in lower
+    ):
+        return "Connection error - check network/endpoint"
+    return None
+
+
+def _is_rate_limit(error_msg: str) -> bool:
+    """True if the error is a throttling/quota response. Being rate-limited
+    proves the request authenticated, so callers treat this as connection-OK.
+    Covers the common phrasings across providers (429, quota, resource
+    exhausted) rather than just the literal words "rate limit"."""
+    lower = error_msg.lower()
+    return (
+        ("rate" in lower and "limit" in lower)
+        or "429" in error_msg
+        or "quota" in lower
+        or "resource has been exhausted" in lower
+        or "resource exhausted" in lower
+    )
+
+
+def _normalize_error_message(error_msg: str) -> Tuple[bool, str]:
+    """Normalize common error patterns into user-friendly messages.
+
+    Used by the *individual model* test, where the user is validating one
+    specific registered model — so a missing model IS a failure (unlike the
+    provider-level test, which only cares that the credentials work).
+    """
+    reason = _connection_failure_reason(error_msg)
+    if reason:
+        return False, reason
+
+    if _is_rate_limit(error_msg):
         return True, "Rate limited - but connection works"
-    elif "not found" in lower and "model" in lower:
+    lower = error_msg.lower()
+    if "not found" in lower and "model" in lower:
         return False, "Model not found on this provider"
-    elif "connection" in lower or "network" in lower:
-        return False, "Connection error - check network/endpoint"
-    elif "timeout" in lower:
-        return False, "Connection timed out - check network/endpoint"
 
     return False, error_msg
 
 
+# Substrings that indicate the provider answered but the *test model* is
+# missing/retired/unsupported - proof the credentials and endpoint work.
+# Only consulted for fixed-endpoint API-key providers (URL-based providers
+# are tested via their own handlers), so a "not found" here is about the
+# model, never a user-supplied base URL.
+_MODEL_UNAVAILABLE_MARKERS = (
+    "not found",
+    "not supported",
+    "does not exist",
+    "deprecated",
+    "unavailable",
+    "no longer available",
+)
+
+
+def classify_provider_test_error(error_msg: str) -> Tuple[bool, str]:
+    """Classify a provider connection-test exception into (success, message).
+
+    The provider test only asks "do these credentials reach a working
+    provider?" - so the sole real failures are a rejected key (401),
+    insufficient permissions (403), and an unreachable endpoint. Anything
+    the provider returned after authenticating - a rate limit, or a
+    missing/retired/unsupported test model - still proves the connection
+    works, so it's reported as success. This is the durable half of the
+    #970 fix: even if the hard-coded test model is retired, a valid key is
+    never misreported as a broken connection.
+    """
+    reason = _connection_failure_reason(error_msg)
+    if reason:
+        return False, reason
+
+    if _is_rate_limit(error_msg):
+        return True, "Rate limited - but connection works"
+    lower = error_msg.lower()
+    if any(marker in lower for marker in _MODEL_UNAVAILABLE_MARKERS):
+        return True, "API key valid (test model unavailable)"
+
+    truncated = error_msg[:100] + "..." if len(error_msg) > 100 else error_msg
+    return False, f"Error: {truncated}"
+
+
 async def test_individual_model(model) -> Tuple[bool, str]:
     """
     Test a specific model configuration end-to-end by making a real API call.

tests/test_connection_test_classification.py

@@ -0,0 +1,137 @@
+"""
+Tests for connection-test error classification (open_notebook/ai/connection_tester.py).
+
+Two semantics share one auth/network classifier:
+- The *provider* test asks only "do these credentials reach a working
+  provider?" — so a missing/retired/unsupported test model is SUCCESS (the
+  #970 durability fix): a hard-coded test model going away must never be
+  misreported as a broken connection.
+- The *individual model* test validates one specific registered model, so a
+  missing model there IS a failure.
+"""
+
+import pytest
+
+from open_notebook.ai.connection_tester import (
+    _connection_failure_reason,
+    _is_rate_limit,
+    _normalize_error_message,
+    classify_provider_test_error,
+)
+
+# Realistic provider error strings.
+GOOGLE_RETIRED_MODEL_404 = (
+    "404 models/gemini-2.0-flash is not found for API version v1beta, "
+    "or is not supported for generateContent."
+)
+GOOGLE_DEPRECATED = "400 Model gemini-1.5-pro has been deprecated."
+GOOGLE_BAD_KEY_401 = "401 API key not valid. Please pass a valid API key."
+GOOGLE_PERM_403 = "403 Permission denied on resource project."
+GOOGLE_QUOTA_429 = "429 Resource has been exhausted (e.g. check quota)."
+DNS_FAILURE = "Connection error: [Errno -2] Name or service not known (getaddrinfo failed)"
+TIMEOUT = "Request timed out after 10s"
+
+
+class TestConnectionFailureReason:
+    """Only auth/permission/network are true connection failures."""
+
+    @pytest.mark.parametrize(
+        "msg,expected_fragment",
+        [
+            (GOOGLE_BAD_KEY_401, "Invalid API key"),
+            (GOOGLE_PERM_403, "lacks required permissions"),
+            (DNS_FAILURE, "Connection error"),
+            (TIMEOUT, "timed out"),
+        ],
+    )
+    def test_real_failures_return_a_reason(self, msg, expected_fragment):
+        reason = _connection_failure_reason(msg)
+        assert reason is not None
+        assert expected_fragment in reason
+
+    @pytest.mark.parametrize(
+        "msg",
+        [GOOGLE_RETIRED_MODEL_404, GOOGLE_DEPRECATED, GOOGLE_QUOTA_429],
+    )
+    def test_provider_reached_returns_none(self, msg):
+        # A model/quota problem came back FROM the provider — not a failure.
+        assert _connection_failure_reason(msg) is None
+
+
+class TestRateLimitDetection:
+    @pytest.mark.parametrize(
+        "msg",
+        [
+            "429 Too Many Requests",
+            "Resource has been exhausted (e.g. check quota).",
+            "You have hit your rate limit",
+            "quota exceeded for this project",
+        ],
+    )
+    def test_detects_throttling_phrasings(self, msg):
+        assert _is_rate_limit(msg) is True
+
+    def test_plain_model_error_is_not_rate_limit(self):
+        assert _is_rate_limit(GOOGLE_RETIRED_MODEL_404) is False
+
+
+class TestClassifyProviderTestError:
+    """Provider test: only bad creds / unreachable endpoint fail."""
+
+    def test_retired_test_model_is_success(self):
+        # The core #970 regression: a shut-down test model must report the
+        # key as valid, not the connection as broken.
+        success, message = classify_provider_test_error(GOOGLE_RETIRED_MODEL_404)
+        assert success is True
+        assert "test model unavailable" in message
+
+    def test_deprecated_model_is_success(self):
+        success, _ = classify_provider_test_error(GOOGLE_DEPRECATED)
+        assert success is True
+
+    def test_rate_limit_is_success(self):
+        success, message = classify_provider_test_error(GOOGLE_QUOTA_429)
+        assert success is True
+        assert "connection works" in message
+
+    @pytest.mark.parametrize(
+        "msg,expected_fragment",
+        [
+            (GOOGLE_BAD_KEY_401, "Invalid API key"),
+            (GOOGLE_PERM_403, "lacks required permissions"),
+            (DNS_FAILURE, "Connection error"),
+            (TIMEOUT, "timed out"),
+        ],
+    )
+    def test_auth_and_network_still_fail(self, msg, expected_fragment):
+        success, message = classify_provider_test_error(msg)
+        assert success is False
+        assert expected_fragment in message
+
+    def test_unrecognized_error_stays_a_failure(self):
+        # A construction/config error we can't attribute to the provider is
+        # surfaced, not silently reported as success.
+        success, message = classify_provider_test_error(
+            "TypeError: create_language() missing required config 'base_url'"
+        )
+        assert success is False
+        assert "Error:" in message
+
+
+class TestIndividualModelSemanticsDiffer:
+    """The SAME model-not-found string is failure for the individual-model
+    test but success for the provider test — the intended difference."""
+
+    def test_model_not_found_is_failure_for_individual_test(self):
+        success, message = _normalize_error_message(GOOGLE_RETIRED_MODEL_404)
+        assert success is False
+        assert "Model not found" in message
+
+    def test_same_string_is_success_for_provider_test(self):
+        success, _ = classify_provider_test_error(GOOGLE_RETIRED_MODEL_404)
+        assert success is True
+
+    def test_individual_test_shares_auth_classification(self):
+        success, message = _normalize_error_message(GOOGLE_BAD_KEY_401)
+        assert success is False
+        assert "Invalid API key" in message

tests/test_credentials_api.py

@@ -269,15 +269,15 @@ def test_classify_matrix(self):
         assert classify_model_type("scribe_v1", "elevenlabs") == "speech_to_text"
         assert classify_model_type("eleven_multilingual_v2", "elevenlabs") == "text_to_speech"
 
-    def test_google_and_vertex_use_current_test_model(self):
-        # Regression test for #970: the connection test used the retired
-        # gemini-2.0-flash, so testing a valid Google AI key failed with 404.
-        # gemini-3.5-flash is the current stable GA (gemini-2.5-flash is
-        # already scheduled for shutdown), so pin the longer-lived one.
+    def test_google_and_vertex_use_floating_alias(self):
+        # Regression test for #970: the connection test used a hard-coded
+        # Gemini id (gemini-2.0-flash) that Google later shut down, so a
+        # valid key failed with 404. Use Google's floating alias, which the
+        # provider repoints on each retirement, so it can't go stale.
         from open_notebook.ai.connection_tester import TEST_MODELS
 
-        assert TEST_MODELS["google"] == ("gemini-3.5-flash", "language")
-        assert TEST_MODELS["vertex"] == ("gemini-3.5-flash", "language")
+        assert TEST_MODELS["google"] == ("gemini-flash-latest", "language")
+        assert TEST_MODELS["vertex"] == ("gemini-flash-latest", "language")
 
 
 if __name__ == "__main__":