Skip to content

AUDIT-2026-04-24.md

Latest commit

 

History

History
124 lines (100 loc) · 5.87 KB

AUDIT-2026-04-24.md

File metadata and controls

124 lines (100 loc) · 5.87 KB
Raw

BenchLoop Audit — 2026-04-24

Current State

Core CLI (bench-loop/)

Working:

  • Full Python CLI with run, compare, submit, info commands
  • 7 suites shipped: speed, toolcall, dataextract, instructfollow, reasonmath, loop_code_fix, loop_schema (93 tasks total)
  • Ollama + OpenAI-compatible providers
  • Hardware detection (CPU, GPU, VRAM, RAM, OS)
  • Run bundle persistence to ~/.bench-loop/runs/
  • Normalized scoring (0–100) with quality/speed/reliability/value aggregates
  • Phase 1+2 revision plan executed: all packs rewritten with original prompts, L*-NN IDs
  • Phase 2 packs (loop_code_fix, loop_schema) built with artifact verification

Stubbed / Incomplete:

  • Harness layer: only raw works. OCPlatform, Hermes, Aider adapters are stubs
  • Phase 3 scoring polish (variance reporting, partial credit unification) — TODO
  • Phase 4 FrankenGPU CLI design — in progress
  • Phase 5 harness matrix — queued
  • No bench-loop submit backend endpoint (CLI can't upload yet)

Web App (bench-loop-app/)

API (FastAPI):

  • Health, hardware, model listing (auto-detects Ollama/LM Studio/oMLX/Jan/vLLM)
  • Ollama version compatibility checks + preflight diagnostics
  • HuggingFace model search + GGUF file size detection
  • Model pull with SSE streaming progress
  • Benchmark run trigger with SSE streaming
  • Run listing from ~/.bench-loop/runs/ on disk
  • Run detail endpoint
  • No submission/upload endpoint — CLI submit has nowhere to POST
  • No leaderboard data endpoint for community results
  • No schemas.py — no Pydantic models for submissions

UI (React + Vite + React Router):

  • 4 tabs: Models, Chat, Benchmark, Leaderboard
  • Models tab: auto-discovers local providers, shows installed models, HF search, pull UI
  • Benchmark tab: run trigger with SSE progress streaming
  • Leaderboard tab: exists but only reads from /api/benchmark/runs (local runs)
    • Groups best score per model, sorts by overall_score
    • Shows Overall / Quality / Speed / Reliability via ScoreBadge
    • No stack-level comparison, no speed data (tps), no community data
  • Chat tab: exists (unclear completeness)
  • No Tailwind — using raw CSS (App.css) with CSS variables
  • No third-party leaderboard data source wired

Results Data (2026-04-24)

  • 3 JSONL files from today's Qwen3.6-27B speed benchmarking:
    • phase1-draft-params.jsonl — 5 config tags, draft param tuning (dmax/p_min variations)
    • phase2-draft-size.jsonl — 5 config tags, draft model size (0.6B vs 1.7B, dmax/p combos)
    • phase3-long-context.jsonl — 6 config tags, context scaling 192k→768k with rope/YaRN
  • Format: {"kind":"run",...} per task + {"kind":"summary",...} per config tag
  • Summary includes: tag, gen_tps_{peak,mean,median,stdev}, n_runs, repeats, max_tokens
  • Not in submission schema format — needs ingestion

Critical Gaps for v0.2 Launch

Gap Impact Effort
No submission schema Can't accept community results 1h
No /api/submit endpoint CLI submit is dead-end 2h
Leaderboard shows local runs only, no stack data Main product value missing 2-3h
No leaderboard.json or equivalent data file Nothing to show on launch 1h (ingest script)
Phase 3 scoring polish incomplete Credibility risk 3-4h
No CI/CD or deploy pipeline for web app Can't ship publicly 2h
No auth or API keys for submissions Spam/junk risk 2-3h

Quick Wins (≤1h each)

  1. Write submission JSON schema + Pydantic models — defines the contract for everything downstream
  2. Ingest existing phase results into leaderboard-v1.json — get real data visible immediately
  3. Wire LeaderboardTab to also read from leaderboard-v1.json — add stack speed columns (peak/mean/median tps)
  4. Add /api/leaderboard endpoint that serves leaderboard-v1.json (static file or DB later)
  5. Add stdev column to leaderboard — shows run consistency, key differentiator

Bigger Items (≥1h each)

  1. /api/submit endpoint — validate against schema, dedup, store canonical row (2h)
  2. Schema migration plan — how to evolve submission-v1 without breaking old data (1-2h)
  3. Auth / API key for submissions — even basic bearer token prevents spam (2h)
  4. Deploy pipeline — Vercel/Fly/Railway for the web app (2h)
  5. Phase 3 scoring polish — variance reporting, partial credit unification (3-4h)
  6. Stack comparison view — side-by-side diff of two stacks (3h)
  7. Hardware filter on leaderboard — GPU/VRAM bucket dropdown (2h)
  8. FrankenGPU design pass — CLI + web green palette refresh (4h+)

Prioritized Todo for Next Week

P0 — Must ship for v0.2

  1. ✅ Submission schema (JSON + Pydantic) — done today
  2. ✅ Ingest phase results → leaderboard-v1.json — done today
  3. Wire leaderboard UI to show stack speed data (peak/mean/stdev tps)
  4. Add /api/leaderboard GET endpoint serving leaderboard data
  5. Add /api/submit POST endpoint with schema validation

P1 — Should ship for credible launch

  1. Basic auth/API key for submit endpoint
  2. Deploy web app to public URL
  3. README update with "submit your results" instructions
  4. Phase 3 scoring polish (variance reporting)
  5. Landing page / hero section explaining what BenchLoop measures

P2 — Nice to have for launch week

  1. Hardware filter dropdown on leaderboard
  2. Stack comparison modal / page
  3. FrankenGPU CLI design pass
  4. Export leaderboard as CSV/markdown
  5. Social sharing for individual stack results

Key Insight

The biggest gap is the data pipeline: schema → ingest → API → UI. The CLI and benchmark engine are solid (93 tasks, 7 suites). The web app has good bones (model discovery, benchmark streaming, run management). The missing piece is the community leaderboard flow — which is the entire product differentiator. Fix the pipeline this week, ship everything else incrementally.