A benchmark using questions from the German version of "Who Wants to Be a Millionaire?". It consists of 45 games, each with 15 questions of increasing difficulty. All tested models played through the 45 games at least once. Each game is ended with the first incorrect answer, and the winnings at that point are recorded. No jokers were used.
The initial questions often involve wordplay and idioms, requiring a deep understanding of the German language. These proved most challenging for LLMs, though they are easily solvable by the average German.
-
Clone the repo or download it as a zip.
git clone https://github.com/ikiruneo/millionaire-bench.git -
Edit the
benchmark.py(lines 14-16) for server configuration. -
Run the benchmark script:
python3 benchmark.py -
View the leaderboard locally at http://127.0.0.1:8020/ by running this inside the project folder:
python3 -m http.server 8020
Check the /wiki/guide for more detailed instructions.
https://millionaire-bench.referi.de/
Human average winnings of the first 999 shows is 36 000€ (source).
My selection of LLMs is limited as I ran them on my Framework Laptop 13 (AMD Ryzen 5 7640U with 32 GB of RAM), which necessitated the use of smaller models. All models were Q4_K_M quantized and, if available, recommended settings were used.
Thanks to the reddit users FullOf_Bad_Ideas and Pauli1_Go for benchmarking cloud models.
Origin of the Q&A's: https://github.com/GerritKainz/wer_wird_millionaer