Skip to content

Commit 0a5f56c

Browse files
fenarfenar
committed
leaderboard updates
1 parent a58d8ed commit 0a5f56c

5 files changed

Lines changed: 303 additions & 40 deletions

File tree

‎benchmarks/model-reports/marathon/kimi-dev-72b.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# kimi-dev-72b - TelcoAIBench Marathon Report
2+
3+
**HF repo:** `abhishekchohan/Kimi-Dev-72B-AWQ`
4+
**Serving:** vLLM v0.26.0 (KServe RawDeployment, 1x RTX PRO 6000 Blackwell 96GB) | args: `--max-model-len=32768`
5+
**Endpoint at test time:** `http://kimi-dev-72b-predictor.gsma-otel2-test.svc.cluster.local:8080`
6+
7+
## Results
8+
9+
| Suite | Accuracy | StdErr | Samples | Tier | Date |
10+
|---|---|---|---|---|---|
11+
| 3gpp | **0.300** | 0.0461 | 100 | lite | 2026-08-12 |
12+
| 6g_bench | **0.607** | 0.04 | 150 | lite | 2026-08-12 |
13+
| oranbench | **0.740** | 0.0359 | 150 | lite | 2026-08-12 |
14+
| srsranbench | **0.820** | 0.0315 | 150 | lite | 2026-08-12 |
15+
| telelogs | **0.340** | 0.0476 | 100 | lite | 2026-08-12 |
16+
| telemath | **0.420** | 0.0496 | 100 | lite | 2026-08-12 |
17+
| teleqna | **0.767** | 0.0134 | 1000 | lite | 2026-08-12 |
18+
| teletables | **0.320** | 0.0469 | 100 | lite | 2026-08-12 |
19+
20+
## Generation behavior
21+
22+
| Suite | Samples | Mean latency (s) | Mean output tokens |
23+
|---|---|---|---|
24+
| 3gpp | 100 | 25.7 | 806 |
25+
| 6g_bench | 150 | 83.6 | 2402 |
26+
| oranbench | 150 | 21.4 | 709 |
27+
| srsranbench | 150 | 24.4 | 791 |
28+
| telelogs | 100 | 108.3 | 3023 |
29+
| telemath | 100 | 230.1 | 6134 |
30+
| teleqna | 1000 | 35.5 | 1049 |
31+
| teletables | 100 | 98.4 | 2869 |
32+
33+
## Sample misses (first per suite, for audit)
34+
35+
- **3gpp** dataset index 0: expected `SA5`, parsed `SA2` - output: "◁think▷Okay, let's try to figure out which 3GPP working group this text belongs to. The user provided a list of working groups, so I need to match the content to one of them. First, the text mentions Non-Public Networks..."
36+
- **3gpp** dataset index 1: expected `RAN4`, parsed `RAN1` - output: "◁think▷Okay, let's try to figure out which 3GPP working group this text belongs to. The user provided a list of working groups, so I need to match the content to one of them. The text mentions "band combination", "inter..."
37+
- **3gpp** dataset index 3: expected `SA5`, parsed `CT1` - output: "◁think▷Okay, let's try to figure out which 3GPP working group this text belongs to. The user provided a list of working groups, so I need to match the content to one of them. Looking at the text, it mentions things like..."
38+
- **6g_bench** dataset index 4: expected `C`, parsed `B` - output: "◁think▷Okay, let's tackle this problem. The user is asking which slice the UAV should switch to based on the current requirements and projected conditions. First, the current situation: the UAV is on mMTC with 25ms late..."
39+
- **6g_bench** dataset index 5: expected `C`, parsed `B` - output: "◁think▷Okay, let's try to figure out which scheduler reconfiguration is best here. The problem is about minimizing maximum regret under worst-case conditions. The user provided several options, each with their own trade-..."
40+
- **6g_bench** dataset index 7: expected `B`, parsed `` - output: "◁think▷Okay, let's try to figure out the best answer here. The problem is about deciding whether to accept a compute offload request from a third party. The UAV is returning under a URLLC slice with certain parameters, a..."
41+
- **oranbench** dataset index 8: expected `B`, parsed `D` - output: "◁think▷Okay, let's tackle this question. The user is asking about the primary purpose of the R1 DME (Data Management Entity) in O-RAN. The options given are: A) To provide a standardized interface for rApps to publish d..."
42+
- **oranbench** dataset index 22: expected `C`, parsed `B` - output: "◁think▷Okay, let's see. The user is asking about the primary goal of the O-RAN ALLIANCE Test and Integration Focus Group (TIFG). The options are: A) To develop and maintain the O-RAN interface specifications. B) To cond..."
43+
- **oranbench** dataset index 25: expected `B`, parsed `A` - output: "◁think▷Okay, let's tackle this problem. The user is asking about the maximum acceptable relative time error between the O-DU and O-RU for S-plane measurement signals in O-RAN. The options given are 1.5 µs, 3 µs, 10 µs, a..."
44+
- **srsranbench** dataset index 0: expected `A`, parsed `C` - output: "◁think▷Okay, let's try to figure out the purpose of the lower_phy_controller class based on the given options. First, the name "lower_phy_controller" suggests it's responsible for controlling the lower physical layer. ..."
45+
46+
## AI Grid tier fit (measured, auto-scored phase)
47+
48+
**Recommended placement: Tier 4 (Core DC)** - 73B params, 40GB awq-4bit, ~78.4s/answer, ~2222 tokens/answer.
49+
Notes: verbose/slow decode - latency envelope only fits a tier up; served quantized (awq-4bit)
50+
51+
| Axis (AIGrid) | Measured accuracy |
52+
|---|---|
53+
| knowledge | 0.776 |
54+
| protocol | 0.453 |
55+
| math | 0.420 |
56+
| fault | 0.340 |
57+
| structured | 0.320 |
58+
59+
*Fitment from measured accuracy, decode speed, verbosity and VRAM - refreshes after the judged pass.*
60+
61+
---
62+
*Per-sample transcripts (full question/answer/verdict JSONL) are archived on the portal state volume under `benchmark-results/`, one directory per suite run, named by model. Scores flow to the [leaderboard](../../docs/data/LEADERBOARD.md) automatically.*

‎docs/data/LEADERBOARD.md‎

Lines changed: 16 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# TelcoAIBench Leaderboard
22

3-
Generated 2026-08-12 20:48 | board judge: `gpt-5.5`
3+
Generated 2026-08-12 23:27 | board judge: `gpt-5.5`
44

55
Composite = importance-weighted mean (weights in `leaderboard_weights.json`). Ranked entries need >= 65% weight coverage; only clean full-set runs are recorded; board judge: `gpt-5.5` - judged scores from other judges are excluded.
66

@@ -16,17 +16,18 @@ Composite = importance-weighted mean (weights in `leaderboard_weights.json`). Ra
1616
| 8 | gpt-oss-20b | **0.5747** | 70% | 0.774 | 0.290 | 0.787 | 0.787 | 0.610 | 0.410 | 0.280 | 0.660 | - | - |
1717
| 9 | mistral-small-3-2-24b | **0.5654** | 70% | 0.760 | 0.290 | 0.760 | 0.773 | 0.390 | 0.240 | 0.530 | 0.780 | - | - |
1818
| 10 | falcon-h1-34b | **0.5514** | 70% | 0.771 | 0.340 | 0.780 | 0.773 | 0.510 | 0.220 | 0.290 | 0.727 | - | - |
19-
| 11 | qwen3-6-35b-a3b | **0.5303** | 70% | 0.812 | 0.330 | 0.827 | 0.807 | 0.510 | 0.150 | 0.080 | 0.727 | - | - |
20-
| 12 | exaone-4-0-32b | **0.5298** | 70% | 0.742 | 0.320 | 0.760 | 0.833 | 0.450 | 0.180 | 0.280 | 0.673 | - | - |
21-
| 13 | r1-distill-qwen-32b | **0.5257** | 70% | 0.739 | 0.300 | 0.727 | 0.800 | 0.470 | 0.320 | 0.250 | 0.600 | - | - |
22-
| 14 | hunyuan-a13b | **0.5221** | 70% | 0.733 | 0.380 | 0.687 | 0.767 | 0.480 | 0.270 | 0.200 | 0.660 | - | - |
23-
| 15 | glm-4-5-air | **0.5007** | 70% | 0.769 | 0.270 | 0.720 | 0.747 | 0.300 | 0.410 | 0.150 | 0.640 | - | - |
24-
| 16 | granite-4-0-h-small | **0.4925** | 70% | 0.690 | 0.270 | 0.733 | 0.680 | 0.330 | 0.250 | 0.300 | 0.687 | - | - |
25-
| 17 | qwen3-6-27b | **0.4818** | 70% | 0.814 | 0.320 | 0.800 | 0.827 | 0.270 | 0.050 | 0.080 | 0.693 | - | - |
26-
| 18 | phi-4-reasoning-plus | **0.4719** | 70% | 0.722 | 0.310 | 0.760 | 0.793 | 0.000 | 0.380 | 0.090 | 0.720 | - | - |
27-
| 19 | otel-llm-20b-it | **0.4615** | 70% | 0.712 | 0.270 | 0.747 | 0.713 | 0.030 | 0.270 | 0.270 | 0.680 | - | - |
28-
| 20 | nemotron-3-nano-30b | **0.4535** | 70% | 0.718 | 0.080 | 0.667 | 0.700 | 0.620 | 0.000 | 0.270 | 0.573 | - | - |
29-
| 21 | eurollm-22b | **0.4515** | 70% | 0.682 | 0.260 | 0.707 | 0.720 | 0.200 | 0.140 | 0.210 | 0.693 | - | - |
30-
| 22 | apertus-8b | **0.3854** | 70% | 0.633 | 0.220 | 0.587 | 0.747 | 0.110 | 0.100 | 0.240 | 0.447 | - | - |
31-
| 23 | eurollm-9b | **0.3653** | 70% | 0.496 | 0.280 | 0.553 | 0.787 | 0.010 | 0.000 | 0.230 | 0.567 | - | - |
32-
| - | kimi-dev-72b | 0.7670 | 9% (unranked) | 0.767 | - | - | - | - | - | - | - | - | - |
19+
| 11 | kimi-dev-72b | **0.5392** | 70% | 0.767 | 0.320 | 0.740 | 0.820 | 0.420 | 0.340 | 0.300 | 0.607 | - | - |
20+
| 12 | qwen3-6-35b-a3b | **0.5303** | 70% | 0.812 | 0.330 | 0.827 | 0.807 | 0.510 | 0.150 | 0.080 | 0.727 | - | - |
21+
| 13 | exaone-4-0-32b | **0.5298** | 70% | 0.742 | 0.320 | 0.760 | 0.833 | 0.450 | 0.180 | 0.280 | 0.673 | - | - |
22+
| 14 | r1-distill-qwen-32b | **0.5257** | 70% | 0.739 | 0.300 | 0.727 | 0.800 | 0.470 | 0.320 | 0.250 | 0.600 | - | - |
23+
| 15 | hunyuan-a13b | **0.5221** | 70% | 0.733 | 0.380 | 0.687 | 0.767 | 0.480 | 0.270 | 0.200 | 0.660 | - | - |
24+
| 16 | glm-4-5-air | **0.5007** | 70% | 0.769 | 0.270 | 0.720 | 0.747 | 0.300 | 0.410 | 0.150 | 0.640 | - | - |
25+
| 17 | granite-4-0-h-small | **0.4925** | 70% | 0.690 | 0.270 | 0.733 | 0.680 | 0.330 | 0.250 | 0.300 | 0.687 | - | - |
26+
| 18 | qwen3-6-27b | **0.4818** | 70% | 0.814 | 0.320 | 0.800 | 0.827 | 0.270 | 0.050 | 0.080 | 0.693 | - | - |
27+
| 19 | phi-4-reasoning-plus | **0.4719** | 70% | 0.722 | 0.310 | 0.760 | 0.793 | 0.000 | 0.380 | 0.090 | 0.720 | - | - |
28+
| 20 | otel-llm-20b-it | **0.4615** | 70% | 0.712 | 0.270 | 0.747 | 0.713 | 0.030 | 0.270 | 0.270 | 0.680 | - | - |
29+
| 21 | nemotron-3-nano-30b | **0.4535** | 70% | 0.718 | 0.080 | 0.667 | 0.700 | 0.620 | 0.000 | 0.270 | 0.573 | - | - |
30+
| 22 | eurollm-22b | **0.4515** | 70% | 0.682 | 0.260 | 0.707 | 0.720 | 0.200 | 0.140 | 0.210 | 0.693 | - | - |
31+
| 23 | apertus-8b | **0.3854** | 70% | 0.633 | 0.220 | 0.587 | 0.747 | 0.110 | 0.100 | 0.240 | 0.447 | - | - |
32+
| 24 | eurollm-9b | **0.3653** | 70% | 0.496 | 0.280 | 0.553 | 0.787 | 0.010 | 0.000 | 0.230 | 0.567 | - | - |
33+
| - | kimi-vl-a3b-thinking | 0.3496 | 61% (unranked) | 0.647 | 0.090 | 0.627 | 0.673 | 0.210 | 0.090 | 0.110 | - | - | - |

‎docs/data/leaderboard.json‎

Lines changed: 131 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"generated": "2026-08-12 20:48",
2+
"generated": "2026-08-12 23:27",
33
"weights": {
44
"teleqna": 1.0,
55
"teletables": 1.0,
@@ -797,6 +797,81 @@
797797
},
798798
"ranked": true
799799
},
800+
{
801+
"key": "kimi-dev-72b",
802+
"model": "kimi-dev-72b",
803+
"endpoint": "http://kimi-dev-72b-predictor.gsma-otel2-test.svc.cluster.local:8080",
804+
"composite": 0.5392,
805+
"coverage": 0.696,
806+
"judge": "-",
807+
"results": {
808+
"teleqna": {
809+
"accuracy": 0.767,
810+
"stderr": 0.0134,
811+
"n": 1000,
812+
"tier": "lite",
813+
"judge": null,
814+
"date": "2026-08-12"
815+
},
816+
"teletables": {
817+
"accuracy": 0.32,
818+
"stderr": 0.0469,
819+
"n": 100,
820+
"tier": "lite",
821+
"judge": null,
822+
"date": "2026-08-12"
823+
},
824+
"oranbench": {
825+
"accuracy": 0.74,
826+
"stderr": 0.0359,
827+
"n": 150,
828+
"tier": "lite",
829+
"judge": null,
830+
"date": "2026-08-12"
831+
},
832+
"srsranbench": {
833+
"accuracy": 0.82,
834+
"stderr": 0.0315,
835+
"n": 150,
836+
"tier": "lite",
837+
"judge": null,
838+
"date": "2026-08-12"
839+
},
840+
"telemath": {
841+
"accuracy": 0.42,
842+
"stderr": 0.0496,
843+
"n": 100,
844+
"tier": "lite",
845+
"judge": null,
846+
"date": "2026-08-12"
847+
},
848+
"telelogs": {
849+
"accuracy": 0.34,
850+
"stderr": 0.0476,
851+
"n": 100,
852+
"tier": "lite",
853+
"judge": null,
854+
"date": "2026-08-12"
855+
},
856+
"3gpp": {
857+
"accuracy": 0.3,
858+
"stderr": 0.0461,
859+
"n": 100,
860+
"tier": "lite",
861+
"judge": null,
862+
"date": "2026-08-12"
863+
},
864+
"6g_bench": {
865+
"accuracy": 0.6067,
866+
"stderr": 0.04,
867+
"n": 150,
868+
"tier": "lite",
869+
"judge": null,
870+
"date": "2026-08-12"
871+
}
872+
},
873+
"ranked": true
874+
},
800875
{
801876
"key": "qwen3-6-35b-a3b",
802877
"model": "qwen3-6-35b-a3b",
@@ -1775,20 +1850,68 @@
17751850
],
17761851
"unranked": [
17771852
{
1778-
"key": "kimi-dev-72b",
1779-
"model": "kimi-dev-72b",
1780-
"endpoint": "http://kimi-dev-72b-predictor.gsma-otel2-test.svc.cluster.local:8080",
1781-
"composite": 0.767,
1782-
"coverage": 0.087,
1853+
"key": "kimi-vl-a3b-thinking",
1854+
"model": "kimi-vl-a3b-thinking",
1855+
"endpoint": "http://kimi-vl-a3b-thinking-predictor.gsma-otel2-test.svc.cluster.local:8080",
1856+
"composite": 0.3496,
1857+
"coverage": 0.609,
17831858
"judge": "-",
17841859
"results": {
17851860
"teleqna": {
1786-
"accuracy": 0.767,
1787-
"stderr": 0.0134,
1861+
"accuracy": 0.647,
1862+
"stderr": 0.0151,
17881863
"n": 1000,
17891864
"tier": "lite",
17901865
"judge": null,
17911866
"date": "2026-08-12"
1867+
},
1868+
"teletables": {
1869+
"accuracy": 0.09,
1870+
"stderr": 0.0288,
1871+
"n": 100,
1872+
"tier": "lite",
1873+
"judge": null,
1874+
"date": "2026-08-12"
1875+
},
1876+
"oranbench": {
1877+
"accuracy": 0.6267,
1878+
"stderr": 0.0396,
1879+
"n": 150,
1880+
"tier": "lite",
1881+
"judge": null,
1882+
"date": "2026-08-12"
1883+
},
1884+
"srsranbench": {
1885+
"accuracy": 0.6733,
1886+
"stderr": 0.0384,
1887+
"n": 150,
1888+
"tier": "lite",
1889+
"judge": null,
1890+
"date": "2026-08-12"
1891+
},
1892+
"telemath": {
1893+
"accuracy": 0.21,
1894+
"stderr": 0.0409,
1895+
"n": 100,
1896+
"tier": "lite",
1897+
"judge": null,
1898+
"date": "2026-08-12"
1899+
},
1900+
"telelogs": {
1901+
"accuracy": 0.09,
1902+
"stderr": 0.0288,
1903+
"n": 100,
1904+
"tier": "lite",
1905+
"judge": null,
1906+
"date": "2026-08-12"
1907+
},
1908+
"3gpp": {
1909+
"accuracy": 0.11,
1910+
"stderr": 0.0314,
1911+
"n": 100,
1912+
"tier": "lite",
1913+
"judge": null,
1914+
"date": "2026-08-12"
17921915
}
17931916
},
17941917
"ranked": false

0 commit comments

Comments
 (0)