runner: add [target].language for per-language HF dataset configs

Evaluating facebook/omnilingual-gaia2 previously meant downloading the parquet
shards by hand, converting them with mt/scripts/data/parquet_to_json.py, and
pointing [target].dataset_root at the result. The runner already had a
HuggingFace auto-download path; the only thing missing was a language dimension.

The dataset's parquet columns (scenario_id, scenario) and HF split (test)
already match what download_hf_dataset() reads. The single mismatch was that the
HF config name was derived from the split alone, while this dataset publishes 40
configs named {language}_{split}.

  [target]
  dataset = "facebook/omnilingual-gaia2"
  language = "spa_Latn"

- config.py: MULTILINGUAL_SPLITS, TargetConfig.language, language-aware
  _normalize_splits so "all" means the four translated capabilities, and
  rejection of language with scenario/dataset_root or a malformed code.
- hf_dataset.py: language parameter; the HF config name is prefixed while the
  split directory keeps its bare name, which is what _infer_result_split relies
  on; cache key suffixed per language.
- cli.py: threaded through run-config, a new run-dataset --language, the
  --dry-run summary, and saved run metadata.

Omitting language leaves every path byte-for-byte unchanged, including the cache
directory name, so existing populated caches stay valid. Regression tests pin
that.

Also fixes a pre-existing bug: the split directory was created before the
download, so a failed or interrupted fetch left a partial directory that the
next run's cache check accepted, silently reusing an incomplete split. The
handler now removes it, and catches BaseException so Ctrl-C cleans up too.

Verified end to end with a pass@3 run on the published dataset: OpenClaw with a
local vLLM Qwen3.6-27B agent and a gpt-oss-120b judge, multilingual judge
prompts, correct per-split reporting, and language recorded in run_config.json.

Author: caciolai

README.md

@@ -13,6 +13,10 @@ Meta Agents Research Environments (ARE) is a platform designed to evaluate AI ag
 Looking for the container-based Gaia2-CLI benchmark stack? Start with
 [gaia2-cli](gaia2-cli).
 
+
+> **New:** [facebook/omnilingual-gaia2](https://huggingface.co/datasets/facebook/omnilingual-gaia2), a
+> multilingual extension of Gaia2! See [Omnilingual-Gaia2](#omnilingual-gaia2) below.
+
 ## Table of Contents
 
 - [Background](#background)
@@ -39,6 +43,7 @@ ARE addresses critical gaps in AI agent evaluation by providing:
 | **[Quick Start](https://facebookresearch.github.io/meta-agents-research-environments/quickstart.html)** | Get up and running with your first scenario in just a few minutes with step-by-step instructions. |
 | **[Gaia2 Evaluation](https://facebookresearch.github.io/meta-agents-research-environments/user_guide/gaia2_evaluation.html)** | Build and evaluate your agents on the Gaia2 benchmark, a comprehensive suite of 800 dynamic scenarios across 10 universes. |
 | **[Gaia2 Blog Post](https://huggingface.co/blog/gaia2)** | Learn more about Gaia2 on the Hugging Face blog. |
+| **[Omnilingual-Gaia2](https://huggingface.co/datasets/facebook/omnilingual-gaia2)** | Multilingual translation of Gaia2, for evaluating agents beyond English. Translation pipeline in [gaia2-cli/mt](gaia2-cli/mt/README.md). |
 | **[Paper](https://ai.meta.com/research/publications/are-scaling-up-agent-environments-and-evaluations/)** | Read the research paper detailing the Gaia2 benchmark and evaluation methodology. |
 | **[Demo](https://huggingface.co/spaces/meta-agents-research-environments/demo)** | [Try the ARE Demo on Hugging Face](https://huggingface.co/spaces/meta-agents-research-environments/demo) — Play around with the agent platform directly in your browser, no installation required! |
 | **[Gaia2 Leaderboard](https://huggingface.co/spaces/meta-agents-research-environments/leaderboard)** | Check the self-published results from Gaia2 Benchmark runs. |
@@ -158,6 +163,41 @@ are-benchmark gaia2-run --hf meta-agents-research-environments/gaia2 \
   --hf_upload my-org/gaia2-results
 ```
 
+### Omnilingual-Gaia2
+
+[facebook/omnilingual-gaia2](https://huggingface.co/datasets/facebook/omnilingual-gaia2) is a
+multilingual extension of Gaia2 — 10 languages across the `execution`, `search`,
+`ambiguity` and `adaptability` splits. The gaia2-cli runner downloads and caches it for
+you: set `[target].language` to pick the language, and
+`[judge].prompt_version = "omnilingual-gaia2"` to enable the multilingual judge prompts,
+without which a correct non-English answer is often scored as a miss.
+
+```bash
+# Copy the annotated template, then edit the language, models and endpoints
+cp gaia2-cli/runner/examples/openclaw_qwen_omnilingual_gaia2_pass3.toml ./my_omnilingual_run.toml
+
+# Validate the config and resolve the scenario selection without launching
+export OPENAI_COMPAT_API_KEY="your-api-key"
+gaia2-runner run-config --config ./my_omnilingual_run.toml --dry-run
+
+# Run the evaluation
+gaia2-runner run-config --config ./my_omnilingual_run.toml
+```
+
+The relevant part of the template:
+
+```toml
+[target]
+dataset = "facebook/omnilingual-gaia2"
+language = "spa_Latn"   # or cmn_Hans, deu_Latn, fra_Latn, ...
+
+[judge]
+prompt_version = "omnilingual-gaia2"
+```
+
+To translate Gaia2 into another language yourself, see
+[gaia2-cli/mt](gaia2-cli/mt/README.md).
+
 ## API
 
 ### Core Concepts

gaia2-cli/mt/README.md

@@ -73,15 +73,34 @@ names must match the server's `--served-model-name`.
 
 ## Evaluate
 
-Point the runner at the translated `dataset_root` and select the Omnilingual-GAIA2 judge
-prompts via `[judge].prompt_version`:
+Either way, select the Omnilingual-GAIA2 judge prompts via
+`[judge].prompt_version = "omnilingual-gaia2"`, then:
 
 ```bash
 gaia2-runner run-config \
-    --config gaia2-cli/runner/examples/omnilingual_gaia2.toml
+    --config gaia2-cli/runner/examples/openclaw_qwen_omnilingual_gaia2_pass3.toml
 ```
 
-See [`runner/examples/omnilingual_gaia2.toml`](../runner/examples/omnilingual_gaia2.toml).
+For the **published** dataset there is nothing to download by hand — set
+`[target].language` and the runner fetches and caches the per-language HuggingFace
+config itself:
+
+```toml
+[target]
+dataset = "facebook/omnilingual-gaia2"
+language = "spa_Latn"
+```
+
+For a corpus **you** just translated with the pipeline above, point at its output
+directory instead:
+
+```toml
+[target]
+dataset_root = "/path/to/omnilingual-gaia2/spa_Latn/data"
+splits = ["execution", "search", "ambiguity", "adaptability"]
+```
+
+See [`runner/examples/openclaw_qwen_omnilingual_gaia2_pass3.toml`](../runner/examples/openclaw_qwen_omnilingual_gaia2_pass3.toml).
 
 ## Data converters
 

gaia2-cli/runner/README.md

@@ -195,7 +195,12 @@ Target selection supports:
 - `scenario = "/path/to/scenario.json"` for one scenario
 - `dataset = "org/name"` to load from HuggingFace (auto-downloads and caches)
 - `dataset_root = "..."` to load from a local directory of scenario JSONs
-- Combine with `splits = "search"`, `splits = ["search", "time"]`, or `splits = "all"`
+- `language = "spa_Latn"` for datasets published with one config per language
+  (e.g. `facebook/omnilingual-gaia2`); the runner loads the `{language}_{split}`
+  config and caches each language separately. Only valid with `dataset`.
+- Combine with `splits = "search"`, `splits = ["search", "time"]`, or `splits = "all"`.
+  With `language`, `"all"` means the four translated capabilities — `execution`,
+  `search`, `ambiguity`, `adaptability` — as there is no translated `time` split.
 - `subset = "/path/to/subset.json"` to limit runs to a manifest
 
 For dataset targets, the runner preserves split subdirectories in the output
@@ -224,6 +229,7 @@ Curated examples:
 - `runner/examples/openclaw_sonnet_gaia2_pass1.toml` — OpenClaw + direct Anthropic Sonnet 4.6, public HuggingFace dataset, pass@1
 - `runner/examples/openclaw_google_gaia2_pass1.toml` — OpenClaw + direct Google AI Studio Gemini 3.1 Pro Preview, public HuggingFace dataset, pass@1
 - `runner/examples/openclaw_gpt54_gaia2_pass1.toml` — OpenClaw + direct OpenAI GPT-5.4, public HuggingFace dataset, pass@1
+- `runner/examples/openclaw_qwen_omnilingual_gaia2_pass3.toml` — Omnilingual-GAIA2: the published multilingual dataset via `language`, OpenClaw + locally served Qwen3.6-27B, `gpt-oss-120b` judge on a second endpoint, multilingual judge prompts, pass@3
 - `runner/examples/template_hermes_openai_compat.toml` — generic Hermes template for custom OpenAI chat-completions-compatible endpoints
 - `runner/examples/template_openclaw_openai_compat.toml` — generic OpenClaw template for custom OpenAI chat-completions-compatible endpoints
 

gaia2-cli/runner/examples/omnilingual_gaia2.toml

@@ -0,0 +1,71 @@
+# Template: Omnilingual-GAIA2 configuration.
+#
+# Two things distinguish this from the other templates:
+#   - `[target].language` selects the per-language HuggingFace config, so the
+#     runner downloads and caches the translated scenarios itself;
+#   - `[judge].prompt_version` selects the multilingual judge prompt overrides,
+#     without which a correct non-English answer is routinely scored as a miss.
+#
+# This template uses locally served models via vllm, serve the model before running, e.g.:
+#   vllm serve Qwen/Qwen3.6-27B --port 8000
+#
+# Then edit the model name(s) and base URL(s) to match your deployment.
+# This has been tested end-to-end with vllm 0.25.1
+
+[target]
+dataset = "facebook/omnilingual-gaia2"
+# One of: cmn_Hans, deu_Latn, fra_Latn, hin_Deva, ind_Latn, ita_Latn, jpn_Jpan, por_Latn, spa_Latn, tur_Latn.
+# Downloads to ~/.cache/gaia2/hf_datasets/, keyed per language (override the location with $GAIA2_HF_CACHE).
+# Omitting `splits` (or setting "all") runs the four translated capabilities: execution, search, ambiguity, adaptability.
+language = "spa_Latn"
+splits = ["execution"]
+
+# Smoke test; remove for the full benchmark.
+limit = 1
+
+
+[agent]
+image = "localhost/gaia2-oc:latest"
+provider = "openai-compat"
+model = "Qwen/Qwen3.6-27B"
+# Containers run with --network=host, so localhost here is the same host the model server is bound to.
+base_url = "http://localhost:8000/v1"
+# vLLM ignores the value but requires the header. Use `api_key_env` instead for endpoints that authenticate.
+api_key = "EMPTY"
+# Thinking enabled, as the paper does for open-weight agents.
+thinking = "high"
+
+[judge]
+provider = "openai-compat"
+# A second server, so judge throughput is independent of agent rollouts:
+#   vllm serve openai/gpt-oss-120b --port 8001 --served-model-name gpt-oss-120b
+#
+# `--served-model-name` matters here.
+# The judge goes through litellm, which treats a leading `openai/`
+# as a provider-routing prefix and strips it before sending.
+# A served name of `openai/gpt-oss-120b` therefore goes out as
+# `gpt-oss-120b` and the server answers 404.
+# The awkward alternative is `model = "openai/openai/gpt-oss-120b"`
+model = "openai/openai/gpt-oss-120b"
+base_url = "http://localhost:8001/v1"
+api_key = "EMPTY"
+# Multilingual checker prompts. Omit for the stock gaia2-core prompts.
+prompt_version = "omnilingual-gaia2"
+# gpt-oss-120b with `low` reasoning is the calibrated reference judge config.
+# `extra_body` is forwarded verbatim to litellm, so this lands as a top-level vllm request field.
+extra_body = { reasoning_effort = "low" }
+
+[run]
+timeout = 1200
+health_timeout = 180
+# Agent and judge have their own servers here, so this is bounded by the agent
+# server's throughput. Raise it while watching that server's queue depth.
+concurrency = 4
+# Long non-Latin rollouts idle between tool calls; the 300s default cuts them
+# off and records them as infra errors.
+idle_timeout = 600.0
+# pass@3 is the headline metric; drop to 1 while iterating to cut cost 3x.
+pass_at = 3
+# Relative paths resolve against this config file
+output_dir = "$HOME/omnilingual_gaia2/results/Qwen3.6-27B__openclaw/spa_Latn"
+log_level = "INFO"