Add retry mechanism for transient LLM failures and completeness tracking

[MaximilianAnzinger]

Summary

• Retry transient LLM failures with exponential backoff in the provider layer (template method pattern: generate() wraps _do_generate() with retry logic). Transient errors (408, 429, 5xx, connection/timeout) are retried up to 4 times with 30s base delay and 5 min cap. Non-transient errors (400, 401, 403, 404, ValueError) propagate immediately.
• Track every missing cell via a completeness report. After runner.run(), get_completeness_report() returns expected/present/skipped/failed counts. Transient failures (retries exhausted) are distinguished from legitimate skips (e.g. distractor metric on true/false questions). A completeness.json is written to the run bundle, and the process exits non-zero when transient failures remain.
• Fix pre-existing lint issues in the changed file set (import sorting, tz-aware datetimes, type hint modernization).

Motivation

A real benchmark run (8 models, 4 quizzes, 9 metrics, 288 expected cells) produced only 279 results. One cell vanished due to a 503 queue timeout — _evaluate_question caught it, logged it, returned None, and the row silently disappeared. The upcoming research sweep (~63,500 calls) cannot tolerate silent data loss.

Configuration

Retry parameters are configurable per evaluator in YAML:

evaluators:
  logos_llama:
    provider: openai_compatible
    model: meta-llama/...
    retry_max_attempts: 5
    retry_base_delay: 60
    retry_max_delay: 300

Parameter	Default	Rationale
retry_max_attempts	4	1 initial + 3 retries
retry_base_delay	30s	First wait ~30s, then ~60s, ~120s, capped at 5 min
retry_max_delay	300s	5 min cap prevents absurd waits

Test plan

• test_transient_503_is_retried — 503 triggers retry, succeeds on second attempt
• test_non_transient_400_not_retried — 400 propagates immediately, no sleep
• test_transient_failure_reports_incomplete — TransientLLMError → complete: false
• test_skipped_error_does_not_fail_completeness — ValueError → skip, complete: true
• All 203 existing tests pass
• ruff, black, mypy clean on changed files

🤖 Generated with Claude Code