Skip to content

AUDIT-2026-05-12.md

Latest commit

 

History

History
60 lines (43 loc) · 4.02 KB

AUDIT-2026-05-12.md

File metadata and controls

60 lines (43 loc) · 4.02 KB
Raw

BenchLoop v0.1 Audit — 2026-05-12

Scope: end-to-end pre-release audit. Code, UX, copy, thesis clarity, ship-blockers.

Thesis clarity

Stated thesis (from SPEC.md): BenchLoop measures both raw model capability and model performance attached to real harnesses, on local hardware, in a reproducible way.

Audit verdict: thesis is sound but the agent loop is the single most differentiated feature and was under-sold in landing + docs copy. Most local-LLM benchmarks score single-shot tool calls; only BenchLoop actually runs the tool execution loop. Landing copy now reflects that.

Issues found

High priority (fixed in this audit)

  • Stale "Six ways" copy on landing page — counted 6 suites but we ship 7 (added agent). Fixed.
  • Suites docs page missing agent — only listed 6 suites. Fixed.
  • OG image references /og-image.png but only og-image.svg exists — broken social cards. Fixed to point at the SVG with width/height hints, plus absolute URL so social platforms can resolve it before DNS is even cut over.
  • Scoring doc missed agent component — added the four-component breakdown (correct_final / efficient / no_hallucinated_tools / all_required_called).

Medium priority (worth doing but not ship-blocking)

  • Local app / is Models tab; no obvious "what is this?" landing — users arriving fresh see a model picker without context. Recommendation: keep / as Models (it's the natural starting point for a power user) but add a one-line tagline + link to the docs in the header.
  • Stacks tab is a leftover concept (context-window leaderboard). Either polish or hide before launch — currently it doesn't get traffic and isn't explained anywhere.
  • TTFT is 0 for OpenAI-compatible endpoints because the provider isn't streaming yet. Documented as a known beta caveat in RELEASE.md + Docs page would benefit from one explicit line about this.
  • Agent suite has 8 tasks — small. Two tasks (agent-02 portfolio math, agent-05 refuse-no-tool) currently fail across every model, which is good signal but readers should know the suite is intentionally tight.

Low priority

  • og-image.svg uses fonts that won't be available on social previews (Inter, JetBrains Mono). For full fidelity convert to PNG via a headless renderer before launch. Falls back gracefully — text just renders in default serif.
  • benchloop run console report Speed prints 0.0 when only the agent suite runs. That's correct (no speed tasks), but a casual reader might think speed is broken. Worth labeling.
  • README mentions coding and tool_use were "deferred" in an old version note — needs a sweep to confirm what's stable.
  • benchloop dashboard command is added but untested against the wheel install. Smoke test before publishing.

What the release looks like right now

  • Core CLI: ✅ wheel + sdist build, install into fresh venv works, all commands respond.
  • Suites: ✅ 7 suites, all runnable. Agent suite verified end-to-end with real tool execution.
  • Scoring: ✅ matches between code, README, docs page.
  • Local app: ✅ all tabs work; serial queue prevents endpoint slamming; auto-poll on detail.
  • Public site: ✅ builds, OG metadata correct, leaderboard shows real seed data.
  • Deploy config: ✅ Cloudflare Pages and Vercel both wired.
  • Domain: ⏳ pending DNS + Pages hookup to bench-loop.com.
  • Logo: ⏳ Eric generating pixel-chip variant.

Recommended ship order

  1. Land the audit fixes (this commit).
  2. Deploy site to Cloudflare Pages staging URL.
  3. Drop in real logo when ready.
  4. Cut DNS to bench-loop.com.
  5. Publish wheel to PyPI.
  6. Tweet with the seed leaderboard numbers.

Caveats to publish in the launch post

  • Beta. Frozen task fixtures are small.
  • TTFT for OpenAI-compatible endpoints is not yet streamed (Ollama TTFT is fine).
  • Some Ollama models do not declare tool support; they fail the agent suite cleanly with a useful error.
  • Benchmarks are serialized per endpoint to avoid local server saturation.
  • This is a practical local benchmark, not an academic eval.