Hi MLPerf Training team,
I'm sharing a public draft of a new proposed standard designed to complement the MLPerf benchmarking ecosystem by introducing a vendor-neutral billing and efficiency metric for LLM evaluation:
📄 Standard draft (v0.1):
https://github.com/huouer/LBBS-Standard/tree/main/docs/LBBS_v0_1_Standard_Draft.pdf
🧩 Motivation
MLPerf focuses on system-level performance (throughput, time-to-train, accuracy).
LBBS focuses on cost-per-task × latency, enabling comparable $·s/task reporting across LLM providers (OpenAI, Anthropic, Google, etc.).
🔍 Why this may be relevant
• MLPerf benchmarks runtime and scaling, LBBS benchmarks billing and deployment cost
• Used together, they offer full economics of inference + training
• Standard is open for 90-day community comment
🗣️ Feedback Thread
huouer/LBBS-Standard#2
Would the MLPerf Training WG be open to reviewing this for potential alignment or cross-reference?
Thanks for your time — happy to answer questions here or in a WG call.
Hi MLPerf Training team,
I'm sharing a public draft of a new proposed standard designed to complement the MLPerf benchmarking ecosystem by introducing a vendor-neutral billing and efficiency metric for LLM evaluation:
📄 Standard draft (v0.1):
https://github.com/huouer/LBBS-Standard/tree/main/docs/LBBS_v0_1_Standard_Draft.pdf
🧩 Motivation
MLPerf focuses on system-level performance (throughput, time-to-train, accuracy).
LBBS focuses on cost-per-task × latency, enabling comparable $·s/task reporting across LLM providers (OpenAI, Anthropic, Google, etc.).
🔍 Why this may be relevant
• MLPerf benchmarks runtime and scaling, LBBS benchmarks billing and deployment cost
• Used together, they offer full economics of inference + training
• Standard is open for 90-day community comment
🗣️ Feedback Thread
huouer/LBBS-Standard#2
Would the MLPerf Training WG be open to reviewing this for potential alignment or cross-reference?
Thanks for your time — happy to answer questions here or in a WG call.