You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/foundations/scenarios.rst
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -29,15 +29,15 @@ Anatomy of a Scenario
29
29
Each scenario in our simulated environment consists of four core components that work together to create a comprehensive testing environment.
30
30
31
31
1. The universe foundation provides a realistic environment populated with various applications such as messaging, email, and weather apps,
32
-
along with their initial data states. This foundation serves as the starting context for the simulation.
32
+
along with their initial data states. This foundation serves as the starting context for the simulation.
33
33
2. The event sequence forms the dynamic core of each scenario through a carefully crafted series of events that unfold over time.
34
-
These events create a dynamic environment through messages from contacts, incoming emails including spam, user prompts and questions, and other relevant real-world occurrences.
34
+
These events create a dynamic environment through messages from contacts, incoming emails including spam, user prompts and questions, and other relevant real-world occurrences.
35
35
3. Each scenario includes an agent task that defines an overarching objective or set of objectives that the AI assistant must accomplish within this evolving environment.
36
-
The task is provided by the user to the agent through the AgentUserInterface.
36
+
The task is provided by the user to the agent through the AgentUserInterface.
37
37
4. Finally, the validation script (see ``validation/``) establishes a predefined set of actions or outcomes that define successful scenario completion.
38
-
The validation can be done using a combination of different validation techniques: state-based, using LLM-as-a-Judge, etc...
39
-
In the platform, we encourage tasks designers to evaluate a model based on its *write* actions, i.e. those that change the environment, rather than only state.
40
-
(It is not desirable to have an agent create a contact and then delete it because it was wrong in the first place).
38
+
The validation can be done using a combination of different validation techniques: state-based, using LLM-as-a-Judge, etc...
39
+
In the platform, we encourage tasks designers to evaluate a model based on its *write* actions, i.e. those that change the environment, rather than only state.
40
+
(It is not desirable to have an agent create a contact and then delete it because it was wrong in the first place).
Copy file name to clipboardExpand all lines: docs/user_guide/gaia2_evaluation.rst
+3Lines changed: 3 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -91,6 +91,7 @@ Gaia2 evaluates seven core agent capabilities with equal weight in the final sco
91
91
Robustness to environment instability, API changes, and random failures. Evaluated on the Gaia2-mini subset.
92
92
93
93
**Dataset Structure:**
94
+
94
95
- **Validation set**: 800 standard scenarios (5×160 per capability) + 320 augmentation scenarios (2×160 mini)
95
96
- **Final scoring**: Each of the 7 capabilities receives equal weight, with global scores reported for test sets to address variance from smaller sample sizes.
96
97
@@ -104,6 +105,7 @@ The Gaia2 evaluation process follows these key steps:
104
105
3. **Submission**: Upload traces and JSON report to the leaderboard
105
106
106
107
**Important Notes:**
108
+
107
109
- The test set is entirely private and reserved for internal validation by Meta/Hugging Face
108
110
- All leaderboard submissions are made using the validation set on a voluntary, non-controlled basis
109
111
- Researchers must respect the benchmark integrity and avoid using validation data for training
@@ -226,6 +228,7 @@ The validation split contains oracle events that enable judge-based evaluation f
226
228
--limit 20
227
229
228
230
**Available Configurations:**
231
+
229
232
- ``mini``: Gaia2-mini (160 scenarios across all capabilities)
description = "Meta Agents Research Environments is a research-driven environment designed to simulate complex, real-life tasks that span several minutes and require multiple steps to be solved. Unlike static simulation environments, this platform introduces a dynamic setting where the state of the environment evolves and new information is continuously integrated."
0 commit comments