fix: make the provider connection test resilient to model retirement - #1035
Merged
Conversation
Two independent hardenings against the #970 class of breakage (a hard-coded Gemini test model getting shut down by Google, which made testing a valid key fail with a 404): - Use Google's floating alias gemini-flash-latest for the Google/Vertex test model instead of a dated id, so a retirement repoints it for us. - Reframe the provider connection test around what an error actually proves: only a rejected key (401), missing permissions (403), or an unreachable endpoint are failures. Anything the provider returns after authenticating - a rate limit, or a missing/retired/unsupported model - still proves the credentials work, so it reports success. Previously this relied on matching the literal phrase 'not found' + 'model', which a differently-worded retirement/deprecation error slipped past. Unifies the auth/network/rate-limit classification (previously duplicated and divergent between connection_tester and credentials_service) into shared helpers. The individual-model test keeps model-not-found as a failure, since there a specific registered model really is broken. Adds classification tests with realistic provider error strings.
This was referenced Jul 10, 2026
11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Google/Vertex "Test Connection" feature broke whenever Google retired the hard-coded test model (#970): the test invoked
gemini-2.0-flash, Google shut it down on its published schedule, and a perfectly valid key then failed with a 404 — misreported to the user as a broken connection / bad credential.PRs #996 and #1027 fixed the immediate breakage by swapping in a newer model id, but that's a treadmill: every Gemini id has a ~9–12 month hard-shutdown lifecycle, so the next retirement re-breaks it. This addresses the root cause two ways (belt and suspenders).
Changes
1. Durable test model (belt). Google/Vertex now test with
gemini-flash-latest, Google's provider-maintained floating alias for the current stable Flash model (resolves togemini-3.5-flashtoday). Google repoints the alias on each retirement, so it can't go stale.2. Reframe the provider test around what an error proves (suspenders). The provider connection test only asks "do these credentials reach a working provider?" So the only real failures are a rejected key (401), missing permissions (403), or an unreachable endpoint. Anything the provider returns after authenticating — a rate limit, or a missing/retired/unsupported model — still proves the credentials work, and now reports success. Previously this leaned on matching the literal phrase
"not found"+"model", which a differently-worded retirement/deprecation error slips past → generic failure. Now even if the alias itself ever breaks, a valid key is never misreported.3. Unify duplicated logic. The auth/network/rate-limit classification was duplicated and divergent between
connection_tester._normalize_error_message(model-not-found → failure) andcredentials_service.test_credential's inlineexceptchain (model-not-found → success). Both now share_connection_failure_reason/_is_rate_limit. The individual-model test deliberately keeps model-not-found as a failure — there a specific registered model really is broken; that semantic difference is now explicit and tested.Scope / safety notes
"not found"here is always about the model, never a user-supplied base URL.Tests
New
tests/test_connection_test_classification.pycovers the classifier with realistic provider error strings (Google retired-model 404, deprecation, 401/403, DNS failure, timeout, 429/quota), and asserts the deliberate provider-vs-individual-model semantic difference. Updated the #970 regression test to pin the floating alias.Follow-up: the root fragility is that we string-match because the provider library doesn't surface typed errors — filed upstream at lfnovo/esperanto to expose structured error classification.
Closes #970.