Add omnilingual-gaia2 to README and support easier evaluations launch

[caciolai]

📋 Summary

• Adds a mention to 🤗 facebook/omnilingual-gaia2 in the README

• Adds [target].language as a new TOML attribute to evaluate 🤗 facebook/omnilingual-gaia2 through the existing HuggingFace auto-download path.

---

Before this change, running the dataset meant doing the data plumbing by hand:

• hf download, then
• mt/scripts/data/parquet_to_json.py to convert the parquet shards, then
• pointing [target].dataset_root at the converted tree.

The dataset's parquet columns are scenario_id + scenario and its HF split is test, exactly what download_hf_dataset() already reads.

The only mismatch was that the runner derived the HF config name from the split alone, while the dataset publishes 40 configs named {language}_{split} (10 languages x 4 capabilities).

Now:

[target]
dataset = "facebook/omnilingual-gaia2"
language = "spa_Latn"

Omitting language leaves every existing code path byte-for-byte unchanged, including the
on-disk cache layout, so current users see no difference.

🎯 Type of Change

• 🐛 Bug fix (non-breaking change which fixes an issue)
• ✨ New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
• 📚 Documentation update
• 🔧 Refactoring (no functional changes, no api changes)
• ⚡ Performance improvement
• 🧪 Test addition or improvement
• 🔨 Build/CI changes

Also fixes one latent bug found along the way (see Incidental fixes).

🏗️ Meta Agents Research Environments Components Affected

• Environment (are/simulation/environment.py, are/simulation/core/, environment setup/configuration)
• UI/GUI (are/simulation/gui/, web interface, client-side components)
• Scenarios (are/simulation/scenarios/, scenario definitions, scenario runner)
• Agents (are/simulation/agents/, agent implementations)
• Documentation (README, docs, comments)
• Build/CI (GitHub Actions, Docker, dependencies)

Note: the template's component list predates gaia2-cli/. This PR is confined to
gaia2-cli/runner (config.py, hf_dataset.py, cli.py, tests, one example) plus
README updates. Nothing under are/simulation/ is touched.

🔗 Related Issues

Follows #104, which added Omnilingual-GAIA2 translation and the multilingual judge prompts.

🧪 Testing Strategy

• Unit Tests: Added/updated unit tests
• Integration Tests: Tested integration with other components
• Manual Testing: Manually verified functionality
• Scenario Testing: Ran specific scenarios to validate changes
• GUI Testing: Tested web interface (if applicable)
• Regression Testing: Verified existing functionality still works

Test Details

Unit tests

249 pass in gaia2-cli/runner (was 225; +24 new).

New tests/test_hf_dataset.py (12 new tests) that works offline, providing coverage for these cases:

• config name built as spa_Latn_search;
• files materialized to <cache>/<language-keyed>/search/;
• backward-compatibility

tests/test_run_config.py (+11 additional tests) covers:

• language round-tripping,
• "all" excluding time,
  rejection of time/scenario/dataset_root/malformed codes, the --dry-run output, and
  run-dataset --language forwarding and metadata.

tests/test_cli_judge_config.py (+1 test) pins language into run_config.json.

End-to-end

A full pass@3 run completed on a GPU node with podman, OpenClaw, a local vLLM serving Qwen/Qwen3.6-27B as the agent (port 8000) and gpt-oss-120b as the judge with reasoning_effort: low (port 8001):

• Cold download resolved the right config and materialized real data:

  gaia2_runner.hf_dataset: Downloading HF dataset facebook/omnilingual-gaia2 (configs: ['spa_Latn_search'])
  gaia2_runner.hf_dataset:   spa_Latn_search: downloading ...
  gaia2_runner.hf_dataset:   spa_Latn_search: 160 scenarios
  

  A second invocation logged Using cached dataset at ... instead of re-downloading.

• Cache layout follows its design:
  ~/.cache/gaia2/hf_datasets/facebook_omnilingual-gaia2_spa_Latn/{execution,search}/.

• pass@3 completed across three runs, with the aggregation and per-split breakdown
  both working:

  Per split:
    execution: 1/1 passed, 0/1 failed, 0/1 errors (100.0%)
  ...
  avg@3: 33.3% ± 47.1%
  pass@3: 100.0% ± 0.0%
  

  The execution label confirms _infer_result_split still resolves the split correctly from the language-keyed cache root.

• run_config.json records the new field, and notably carries no time split:

  "dataset": "facebook/omnilingual-gaia2",
  "dataset_cache_dir": ".../hf_datasets/facebook_omnilingual-gaia2_spa_Latn",
  "language": "spa_Latn",
  "splits": ["execution"]

• Error path: a nonexistent language produced a UsageError naming the config it tried,
  listing all 40 available configs, and leaving no partial cache directory.

🤖 AI Usage Declaration

Written with Claude Code (Opus 5) throughout: codebase exploration, the implementation plan,
the code and tests, and this description.

Author reviewed the code, ran the end-to-end evaluation, and edited this PR extensively.

📸 Screenshots/Recordings

N/A — no UI changes.

⚠️ Breaking Changes

None. language is optional, and every branch it touches falls back to current behaviour when it is absent:

• The cache directory suffix is appended conditionally, so existing populated caches under ~/.cache/gaia2/hf_datasets/ stay valid and no one re-downloads.
• HF config names are unprefixed without a language, so non-omnilingual datasets are unaffected.
• Split defaults still expand to all five CANONICAL_SPLITS (including time) with no language.

A dedicated regression test pins each of these.

Incidental fixes, addressed here

Partial cache directories after a failed or interrupted download.

A failed fetch left a directory that the next run accepted, silently resolving zero (or a truncated set of) scenarios for that split.
The download is now wrapped so the directory is removed on failure, and the error names the config and lists what is available.
The handler catches BaseException, not Exception, specifically so that a Ctrl-C mid-download also cleans up rather than leaving a half-populated split cached and KeyboardInterrupt is re-raised untouched.

Pre-existing; not specific to this feature.

Known limitations, not addressed here

Three pre-existing issues surfaced while testing. All are out of scope; each is worth its own
issue.

openai/-prefixed model names are unreachable for the judge.

judge/engine.py:60 prefixes openai/ for openai-compat providers so litellm uses the OpenAI transport, and skips that when the name already starts with openai/.

But litellm strips that prefix as provider routing — so a model genuinely served as openai/gpt-oss-120b goes out asgpt-oss-120b and 404s.
The workaround is model = "openai/openai/gpt-oss-120b", which is what the example config uses and what therefore lands verbatim in run metadata.

Implementing a more robust in-code patch would be better for UX.

Ctrl-C leaves orphaned containers.

Containers are started detached without --rm (launcher.py:440) and removed in a finally inside the worker thread (runner.py:350).
KeyboardInterrupt reaches only the main thread, so those blocks never run and the containers keep driving their agent loops against the model server.
Cleanup is manual: podman rm -f $(podman ps -aq --filter name=gaia2-).

A SIGINT handler that stops live containers would fix it.

run-dataset in HF mode globs the entire cache directory

This is different from run-config, that restricts to the selected splits.

With per-language cache dirs this stays language-correct, but --language spa_Latn --splits search will still pick up other splits previously cached for that language.

📋 Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published

[RomainFrog]

LGTM!