A self-contained portal & benchmark suite to measure any telco AI model - chat with it, watch it, benchmark it.
Landing Page | 
Article | 
Benchmark Suites | 
Verification Report
Point it at any OpenAI-compatible endpoint (vLLM, RHOAI/KServe, TGI, SaaS) with two environment variables, and you get three things:
| Portal |
 Benchmark Suite |
Receipts |
|---|---|---|
| Expert telco chat personas and a live vLLM observability dashboard - persistent sessions, streaming, file upload. | The 8 Open-Telco benchmarks (TeleQnA, TeleTables, TeleMath, TeleLogs, 3GPP-TSG, ORANBench, srsRANBench, 6G-Bench) plus 2 LLM-as-judge suites (Telco's Last Exam, Vendor GenAI deep-dives) with datasets embedded in this repo - run from the UI or CLI, no external dependencies, ever. | Scoring parity-validated against the official GSMA harness (≤1pp on all 7 leaderboard tasks), plus a full leaderboard-claim verification report showing why pinned, reproducible evals matter. |
# 1. serve a model anywhere (example: vLLM)
vllm serve <your-model> --port 8080
# 2. point TelcoAIBench at it - no source edits
pip install 'gradio>=5,<6' && pip install -r requirements-v2.txt
export SME_API_ENDPOINT="https://my-model-route.apps.mylab" # base URL, no /v1
export SME_MODEL_NAME="my-served-model-name"
export SME_TLS_VERIFY="false" # lab self-signed certs
python sme-web-ui-v2.py # :30180 | login admin/minad
# 3. or benchmark from the CLI (identical engine to the Benchmark tab)
cd benchmarks/open-telco
python3 otel_eval.py --endpoint https://<model-route>/v1 --model <name>All configuration variables
| Variable | Purpose | Default |
|---|---|---|
SME_API_ENDPOINT |
OpenAI-compatible base URL (no /v1) |
- |
SME_MODEL_NAME |
served model name | - |
SME_API_TOKEN / SME_USE_TOKEN_AUTH |
bearer auth | true |
SME_TLS_VERIFY |
TLS verification | false |
SME_ADMIN_USERNAME / SME_ADMIN_PASSWORD |
portal login | admin / minad |
SME_STATE_DIR |
home for all mutable state: endpoint + judge registries, chat sessions, prompt personas, metrics archive, benchmark transcripts. Point it at a persistent volume (e.g. a PVC mounted at /data) and everything survives pod restarts. |
app dir |
Kubernetes/OpenShift: a minimal Deployment that clones this repo, pip-installs,
and sets the SME_* env vars is all it takes - plus a Service and
Route/Ingress on port 30180. The Benchmark tab works out of the box because
the datasets ship inside the repo.
Chat - expert telco conversations
Multi-persona chat (Telco / Network / Cloud / Storage experts, intent classification, or your own), persistent shareable sessions, auto-streaming for large contexts, live temperature/token controls, and document upload (txt/md/csv/json/py/pdf).
Prompt Manager - persona engineering
Create, edit, and persist system-prompt personas (system_prompts.json)
without touching code - instantly available in Chat.
Observability - live vLLM metrics
Dual-API dashboard polling the model server's /metrics: request rates,
latency, token throughput, cache utilization, health, efficiency analysis,
and diagnostics - with Plotly visualizations.
Benchmark - leaderboard-grade evals, one click
Pick benchmarks, tier (lite = leaderboard default, or full), sample limit, parallelism, and token cap - results stream in live with per-task progress and running accuracy, ending in accuracy ± stderr per benchmark, the overall average, and per-sample transcripts for auditing.
Multi-model, side by side. Provision any OpenAI-compatible endpoint from
the UI (base URL plus optional API token); every discovered model gets its
own target card, added and removed dynamically and persisted in
benchmark_endpoints.json. Each card runs independently - up to two in
parallel - with its own live results table and a Stop button that
hard-aborts the run: queued samples are dropped and in-flight generations
are cancelled on the server within seconds, with partial results reported
honestly as "stopped (partial)". Runs whose browser page closes are
auto-cancelled too - no orphaned GPU load.
Judged suites. telcos_last_exam (a 30-question telecom exam across
8 domains and 3 difficulty tiers, graded against an official answer key
with per-question grading notes, points-weighted) and vendor_genai
(a 6-vendor x 4-domain deep-dive matrix graded per-criterion: accuracy,
honesty, completeness, depth - weighted so confident fabrication is
punished hardest) have no deterministic scorer - a judge model you
provision grades every answer and returns structured JSON. Use the
dedicated Provision judge model form (endpoint URL + API key + model
name - leave the name empty and the endpoint's models are discovered into
a dropdown); judge endpoints never become benchmark target cards and
persist on the state volume.
Breakdowns and failure reports. Every judged run ends with per-domain, per-difficulty, per-vendor and per-criterion breakdowns inline, plus a downloadable run report (HTML + markdown) listing every question worst-first with its score, verdict, what was missed, and the judge's written rationale.
Leaderboard. Every clean, full-set run is recorded automatically into a
persistent leaderboard (state volume). The Leaderboard tab ranks models
by an importance-weighted composite (judged suites weigh heaviest; weights
editable in leaderboard_weights.json) with three honesty rules: >= 70%
weight coverage to rank, full-set runs only, and one consistent judge -
scores from other judges are excluded and flagged. Publish snapshot
exports leaderboard.json + LEADERBOARD.md; commit them to
docs/data/ and the landing page
renders the public board from the versioned snapshot.
All benchmark assets live under benchmarks/:
| Suite | What it is |
|---|---|
open-telco/ |
Self-contained Open-Telco eval framework - 8 GSMA telecom benchmarks, lite + full datasets embedded (~4.5MB gzipped JSONL), single-file runner (stdlib + requests). Parity-validated; includes leaderboard claim snapshots and the 2026-08 verification report. |
vendor-genai-tests/ |
Vendor GenAI matrix - 24 deep-dives across 6 vendors (Ericsson, Nokia, Mavenir, Samsung, Rakuten Symphony, Cisco) x 4 domains; per-criterion LLM-as-judge with fact anchors, fabrication bait, and honesty traps. |
telcos-last-exam/ |
Telco's Last Exam - 30 questions, 8 domains, 3 difficulty tiers, 246 points; LLM-as-judge against machine-verified answer keys with grading notes; points-weighted scoring. |
model-reports/ |
Per-model benchmark answers and performance reports. |
embeddings/ |
Embeddings model benchmark notes. |
Reproducibility discipline - publish every number with: model revision hash, serving stack + version, precision, temperature, dataset tier, sample counts, and date. The verification report documents exactly what happens when leaderboards skip this.
telcoaibench/
├── sme-web-ui-v2.py # The portal (Gradio), all tabs incl. Benchmark
├── system_prompts.json # Expert persona definitions
├── requirements-v2.txt # Python dependencies (gradio pinned <6)
├── benchmarks/ # All benchmark & eval assets
│ ├── open-telco/ # embedded eval framework: runner + datasets + reports
│ ├── vendor-genai-tests/ # Ericsson / Nokia / Mavenir + Telco5G
│ ├── telcos-last-exam/ # telco exam + per-model answers
│ ├── model-reports/ # per-model results & perf reports
│ └── embeddings/ # embeddings benchmark
├── archive/ # Legacy v1 application
└── images/ # Logo & screenshots
Architecture notes
Single-file app (sme-web-ui-v2.py) with clean separations: Config
(env-var-driven, pluggable endpoint) | ChatClient (OpenAI-compatible HTTP
with smart streaming, retries, timeouts) | SessionManager (file-backed,
24h retention) | MetricsCollector (/metrics polling + Plotly) |
ChatInterface (Gradio UI; the Benchmark tab imports
benchmarks/open-telco/otel_eval.py directly).
Benchmark engine: SSE streaming by default (survives proxy/router idle timeouts on long generations), deterministic scoring ported 1:1 from the official harness, 8k default token cap against runaway chain-of-thought, zero network dependencies for datasets.
Graduated from the telco-sme experiment in
Telco-AIX, where its full
development history lives. Contributions welcome - MIT licensed.