Skip to content

fix: make the provider connection test resilient to model retirement - #1035

Merged
lfnovo merged 2 commits into
mainfrom
fix/connection-test-durable-model-and-semantics
Jul 10, 2026
Merged

fix: make the provider connection test resilient to model retirement#1035
lfnovo merged 2 commits into
mainfrom
fix/connection-test-durable-model-and-semantics

Conversation

@lfnovo

@lfnovo lfnovo commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Problem

The Google/Vertex "Test Connection" feature broke whenever Google retired the hard-coded test model (#970): the test invoked gemini-2.0-flash, Google shut it down on its published schedule, and a perfectly valid key then failed with a 404 — misreported to the user as a broken connection / bad credential.

PRs #996 and #1027 fixed the immediate breakage by swapping in a newer model id, but that's a treadmill: every Gemini id has a ~9–12 month hard-shutdown lifecycle, so the next retirement re-breaks it. This addresses the root cause two ways (belt and suspenders).

Changes

1. Durable test model (belt). Google/Vertex now test with gemini-flash-latest, Google's provider-maintained floating alias for the current stable Flash model (resolves to gemini-3.5-flash today). Google repoints the alias on each retirement, so it can't go stale.

2. Reframe the provider test around what an error proves (suspenders). The provider connection test only asks "do these credentials reach a working provider?" So the only real failures are a rejected key (401), missing permissions (403), or an unreachable endpoint. Anything the provider returns after authenticating — a rate limit, or a missing/retired/unsupported model — still proves the credentials work, and now reports success. Previously this leaned on matching the literal phrase "not found" + "model", which a differently-worded retirement/deprecation error slips past → generic failure. Now even if the alias itself ever breaks, a valid key is never misreported.

3. Unify duplicated logic. The auth/network/rate-limit classification was duplicated and divergent between connection_tester._normalize_error_message (model-not-found → failure) and credentials_service.test_credential's inline except chain (model-not-found → success). Both now share _connection_failure_reason / _is_rate_limit. The individual-model test deliberately keeps model-not-found as a failure — there a specific registered model really is broken; that semantic difference is now explicit and tested.

Scope / safety notes

  • URL-based providers (ollama, openai_compatible, azure) are tested by their own handlers and return before this path, so "not found" here is always about the model, never a user-supplied base URL.
  • Genuinely unrecognized errors (e.g. a client-side construction/config error) still report failure — the reframe is bounded so a config problem isn't silently called success.

Tests

New tests/test_connection_test_classification.py covers the classifier with realistic provider error strings (Google retired-model 404, deprecation, 401/403, DNS failure, timeout, 429/quota), and asserts the deliberate provider-vs-individual-model semantic difference. Updated the #970 regression test to pin the floating alias.

Follow-up: the root fragility is that we string-match because the provider library doesn't surface typed errors — filed upstream at lfnovo/esperanto to expose structured error classification.

Closes #970.

Review in cubic

lfnovo added 2 commits July 10, 2026 16:22
Two independent hardenings against the #970 class of breakage (a hard-coded
Gemini test model getting shut down by Google, which made testing a valid
key fail with a 404):

- Use Google's floating alias gemini-flash-latest for the Google/Vertex
  test model instead of a dated id, so a retirement repoints it for us.
- Reframe the provider connection test around what an error actually
  proves: only a rejected key (401), missing permissions (403), or an
  unreachable endpoint are failures. Anything the provider returns after
  authenticating - a rate limit, or a missing/retired/unsupported model -
  still proves the credentials work, so it reports success. Previously this
  relied on matching the literal phrase 'not found' + 'model', which a
  differently-worded retirement/deprecation error slipped past.

Unifies the auth/network/rate-limit classification (previously duplicated
and divergent between connection_tester and credentials_service) into
shared helpers. The individual-model test keeps model-not-found as a
failure, since there a specific registered model really is broken.

Adds classification tests with realistic provider error strings.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Google AI connection test checks for removed gemini-2.0-flash

1 participant