Skip to content

fix(settings): reliable key validation + deepseek/nvidia probes#21

Open
cmullins70 wants to merge 1 commit into
vasu-devs:mainfrom
cmullins70:fix-provider-key-validation
Open

fix(settings): reliable key validation + deepseek/nvidia probes#21
cmullins70 wants to merge 1 commit into
vasu-devs:mainfrom
cmullins70:fix-provider-key-validation

Conversation

@cmullins70

Copy link
Copy Markdown

Summary

The "Check keys" button in Settings was unreliable for Anthropic and silent for DeepSeek/NVIDIA. This PR fixes both.

What was wrong

Anthropic probe (backend/main.py:1200-1214 on main) was sending a real POST /v1/messages inference call with:

  • a hardcoded model claude-haiku-4-5-20251001 (returns 404 for keys whose tier doesn't include that exact model → mapped to unreachable),
  • a 1-token request body (costs tokens just to validate a key),
  • a 5s total timeout (TLS cold-start on slow networks pushes past this → unreachable).

For a freshly-created key with $0 credit balance, the result was a "broken" badge on a perfectly valid key.

DeepSeek and NVIDIA are first-class providers in the Settings UI (src/settings/shared.tsx:40-47), but the validator's probe-set was hardcoded as {anthropic, openai, groq}. Users with valid keys saw a permanent neutral "unchecked" badge with no signal whether their key worked.

What changed

  • All providers now probe GET /v1/models — free, fast, no model-tier dependency. Anthropic uses x-api-key + anthropic-version: 2023-06-01; OpenAI/Groq/DeepSeek/NVIDIA use Bearer auth. Endpoint shape verified against the official Anthropic SDK.
  • Timeout bumped to 10s total / 5s connect (covers first-request TLS handshake).
  • 403 now maps to invalid_key alongside 401 (revoked/restricted keys are auth failures, not network issues).
  • Added probes for DeepSeek (api.deepseek.com/v1/models) and NVIDIA (integrate.api.nvidia.com/v1/models).
  • Probe-set centralised in a probeable set so _probe_provider_key and validate_settings can't drift.

Tests

New TestProviderKeyProbe class in backend/tests/test_api.py (8 cases, mocks httpx.AsyncClient):

  • Per-provider: correct URL + correct auth header shape (x-api-key vs Authorization: Bearer).
  • 401 and 403invalid_key.
  • 5xxunreachable.
  • Explicit assertion that the probe never POSTs — locks in the original bug.

uv run python -m pytest tests/109 passed, no regressions.

Verification

Manually verified in the desktop app via npm run tauri dev against live keys for OpenAI, Anthropic, DeepSeek, and Groq. All four now go green on first click.

Checklist

  • Ran pytest backend/tests/ — 109 passed.
  • Added tests for the behavior change.
  • Kept the PR focused (single logical change: key-validation probe).
  • No local app data, secrets, or credentials in the diff.
  • No experimental/auto-apply changes.
  • User impact described above.

…robes

The Anthropic probe was hitting POST /v1/messages with a hardcoded model
("claude-haiku-4-5-20251001"), a real (1-token) inference body, and a 5s
timeout. Newly-created keys whose tier didn't include that model returned
404 (mapped to "unreachable"), TLS cold-start could exceed 5s on slow
networks, and the call cost tokens just to validate.

DeepSeek and NVIDIA were configurable in the UI but the validator
hardcoded the probe-set as {anthropic, openai, groq}, so users with valid
DeepSeek/NVIDIA keys saw a permanent "unchecked" badge.

Changes
- All providers now probe GET /v1/models (free, fast, no model-tier
  dependency). Anthropic uses x-api-key + anthropic-version; the rest
  use Bearer auth.
- Timeout: 10s total, 5s connect (covers TLS handshake on first probe).
- 403 (revoked/restricted key) now maps to invalid_key alongside 401,
  rather than unreachable.
- Added DeepSeek probe -> https://api.deepseek.com/v1/models
- Added NVIDIA probe -> https://integrate.api.nvidia.com/v1/models
- Probe-set centralised in a `probeable` set so the two locations stay
  in sync.

Tests
- New TestProviderKeyProbe class (8 cases) covering URL/header shape per
  provider, 401/403 -> invalid_key, 5xx -> unreachable, and an explicit
  assertion that the probe never POSTs (the original bug).
- Full suite still green (109 passed).

Verified manually in the desktop app with live keys for OpenAI,
Anthropic, DeepSeek, and Groq -- all four now go green on first click.
@vercel

vercel Bot commented May 7, 2026

Copy link
Copy Markdown

@cmullins70 is attempting to deploy a commit to the Vasu-Devs' projects Team on Vercel.

A member of the Team first needs to authorize it.

@vasu-devs vasu-devs self-requested a review as a code owner May 14, 2026 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant