Skip to content

Commit 66785e9

Browse files
committed
add bibtex citation to README and update some docs
1 parent 251e23f commit 66785e9

10 files changed

Lines changed: 112 additions & 38 deletions

File tree

are/simulation/gui/client/src/utils/const.tsx

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,7 @@ export const HF_ORG_URL: string =
2020
export const HF_DATASET_URL: string =
2121
"https://huggingface.co/datasets/meta-agents-research-environments/gaia2";
2222

23-
export const ARXIV_URL: string =
24-
"https://ai.meta.com/research/publications/are-scaling-up-agent-environments-and-evaluations/"; // TODO replace with arxiv url
23+
export const ARXIV_URL: string = "https://arxiv.org/abs/2509.17158";
2524

2625
export const DOC_URL: string =
2726
"https://facebookresearch.github.io/meta-agents-research-environments/";

docs/api_reference/events.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ The environment manages events through two main data structures:
8080
env.schedule(event)
8181
8282
# Access event history
83-
completed_events = env.event_log
83+
completed_events = env.event_log.list_view()
8484
8585
Event Graph Nodes
8686
-----------------

docs/foundations/notifications.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,4 +63,4 @@ exploration of how notification timing, prioritization, and context integration
6363

6464

6565
**Next Steps**
66-
* Keep reading the Foundations guide to learn more about :doc:`universes`.
66+
* Keep reading the Foundations guide to learn more about :doc:`scenarios`.

docs/foundations/scenarios.rst

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -29,15 +29,15 @@ Anatomy of a Scenario
2929
Each scenario in our simulated environment consists of four core components that work together to create a comprehensive testing environment.
3030

3131
1. The universe foundation provides a realistic environment populated with various applications such as messaging, email, and weather apps,
32-
along with their initial data states. This foundation serves as the starting context for the simulation.
32+
along with their initial data states. This foundation serves as the starting context for the simulation.
3333
2. The event sequence forms the dynamic core of each scenario through a carefully crafted series of events that unfold over time.
34-
These events create a dynamic environment through messages from contacts, incoming emails including spam, user prompts and questions, and other relevant real-world occurrences.
34+
These events create a dynamic environment through messages from contacts, incoming emails including spam, user prompts and questions, and other relevant real-world occurrences.
3535
3. Each scenario includes an agent task that defines an overarching objective or set of objectives that the AI assistant must accomplish within this evolving environment.
36-
The task is provided by the user to the agent through the AgentUserInterface.
36+
The task is provided by the user to the agent through the AgentUserInterface.
3737
4. Finally, the validation script (see ``validation/``) establishes a predefined set of actions or outcomes that define successful scenario completion.
38-
The validation can be done using a combination of different validation techniques: state-based, using LLM-as-a-Judge, etc...
39-
In the platform, we encourage tasks designers to evaluate a model based on its *write* actions, i.e. those that change the environment, rather than only state.
40-
(It is not desirable to have an agent create a contact and then delete it because it was wrong in the first place).
38+
The validation can be done using a combination of different validation techniques: state-based, using LLM-as-a-Judge, etc...
39+
In the platform, we encourage tasks designers to evaluate a model based on its *write* actions, i.e. those that change the environment, rather than only state.
40+
(It is not desirable to have an agent create a contact and then delete it because it was wrong in the first place).
4141

4242
Evaluation Goals
4343
~~~~~~~~~~~~~~~~

docs/tutorials/apps_tutorial.rst

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
**File**: ``are/simulation/scenarios/scenario_apps_tutorial/scenario.py``
1313

1414
**Concepts Covered**:
15+
1516
- Data model design with dataclasses and enums
1617
- App class structure and inheritance patterns
1718
- Tool method registration and decorators
@@ -108,6 +109,7 @@ benchmarks.
108109
4. **Integration Layer**: Event registration and environment hooks
109110

110111
**Key Takeaways**:
112+
111113
- How to design robust data models with proper validation
112114
- App lifecycle management (initialization, state, reset)
113115
- Tool registration patterns and decorator usage

docs/tutorials/scenario_development.rst

Lines changed: 31 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -51,13 +51,13 @@ First, create a class that inherits from the ``Scenario`` base class and decorat
5151
"""
5252
A custom scenario that demonstrates email and calendar integration.
5353
"""
54-
scenario_id: str = "my_custom_scenario"
5554
start_time: float | None = 0
5655
duration: float | None = 600 # Duration in seconds
5756
5857
def init_and_populate_apps(self):
5958
"""Initialize and populate apps with data."""
6059
# Create apps
60+
aui = AgentUserInterface()
6161
email_app = EmailClientApp()
6262
calendar_app = CalendarApp()
6363
@@ -72,14 +72,14 @@ First, create a class that inherits from the ``Scenario`` base class and decorat
7272
)
7373
7474
# Add calendar events
75-
calendar_app.add_calendar_event(
75+
calendar_app.add_calendar_event(
7676
title="Team Standup"
7777
start_datetime="2024-01-15 09:00:00"
7878
end_datetime="2024-01-15 09:30:00"
79-
)
79+
)
8080
8181
# Register apps
82-
self.apps = [email_app, calendar_app]
82+
self.apps = [email_app, calendar_app, aui]
8383
8484
def build_events_flow(self):
8585
"""Define the sequence of events in the scenario."""
@@ -493,37 +493,45 @@ Here's a complete example of a scenario package structure:
493493
494494
# Add sample emails
495495
self.email_app.add_email(
496-
from_addr="boss@company.com",
497-
subject="Urgent: Project Deadline",
498-
body="The project deadline has been moved up. Please confirm receipt."
496+
email=Email(
497+
sender="boss@company.com",
498+
recipients=[self.email_app.user_email],
499+
subject="Urgent: Project Deadline",
500+
content="The project deadline has been moved up. Please confirm receipt.",
501+
),
499502
)
500503
501504
self.email_app.add_email(
502-
from_addr="spam@marketing.com",
503-
subject="Amazing Deal! Click Now!",
504-
body="Limited time offer! Buy now and save 90%!"
505+
email=Email(
506+
sender="spam@marketing.com",
507+
recipients=[self.email_app.user_email],
508+
subject="Amazing Deal! Click Now!",
509+
content="Limited time offer! Buy now and save 90%!",
510+
),
505511
)
506512
507513
self.apps = [self.email_app]
508514
509515
def build_events_flow(self):
510-
# Agent should read and respond to urgent email
511-
self.add_event(
512-
app_name="EmailClientApp",
513-
function_name="send_email",
514-
parameters={
515-
"to": "boss@company.com",
516-
"subject": "Re: Urgent: Project Deadline",
517-
"body": "Received and understood. Will adjust timeline accordingly."
518-
},
519-
predecessor_event_ids=[],
520-
event_type=EventType.AGENT
521-
)
516+
aui = self.get_typed_app(AgentUserInterface)
517+
email_client = self.get_typed_app(EmailClientApp)
518+
519+
with EventRegisterer.capture_mode():
520+
# User asks agent to read emails
521+
event1 = aui.send_message_to_agent(
522+
content="Read my emails and respond to the urgent one."
523+
).depends_on(None, delay_seconds=5)
524+
# Agent respond to urgent email
525+
event2 = email_client.send_email(
526+
recipients=["boss@company.com"],
527+
subject="Re: Urgent: Project Deadline",
528+
body="Received and understood. Will adjust timeline accordingly.",
529+
).oracle().depends_on(event1, delay_seconds=5)
522530
523531
def validate(self, env) -> ScenarioValidationResult:
524532
try:
525533
email_app = env.get_app("EmailClientApp")
526-
sent_emails = email_app.get_sent_emails()
534+
sent_emails = email_app.folders[EmailFolderName.SENT].emails
527535
528536
# Check if response was sent
529537
success = any("Re: Urgent" in email.subject for email in sent_emails)

docs/tutorials/working_with_scenarios.rst

Lines changed: 64 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,18 +47,20 @@ Every Meta Agents Research Environments scenario follows a consistent structure
4747
# 1. Initialize applications
4848
email_app = EmailClientApp()
4949
calendar_app = CalendarApp()
50+
messaging_app = MessagingApp()
5051
5152
# 2. Populate initial data
5253
email_app.add_email(...)
5354
calendar_app.add_event(...)
5455
5556
# 3. Register apps with scenario
56-
self.apps = [email_app, calendar_app]
57+
self.apps = [email_app, calendar_app, messaging_app]
5758
5859
def build_events_flow(self) -> None:
60+
messaging_app = self.get_typed_app(MessagingApp)
5961
# 4. Define dynamic events that occur during execution
6062
event1 = Event.from_function(
61-
messaging.add_message,
63+
messaging_app.add_message,
6264
conversation_id="conv1",
6365
sender="John Doe",
6466
content="Hello!"
@@ -171,6 +173,66 @@ Validation functions determine scenario success. They typically check:
171173
Common Validation Patterns
172174
~~~~~~~~~~~~~~~~~~~~~~~~~~
173175

176+
177+
**Validation Event**
178+
179+
.. code-block:: python
180+
181+
def both_responses_validator(env: AbstractEnvironment) -> bool:
182+
"""Check if agent gave both responses."""
183+
aui_app = env.get_app("AgentUserInterface")
184+
agent_messages = aui_app.get_all_messages_from_agent()
185+
has_correct = any(msg.content == "I am a robot" for msg in agent_messages)
186+
has_incorrect = any(
187+
msg.content == "I am not a robot •`_´• " for msg in agent_messages
188+
)
189+
return has_correct and not has_incorrect
190+
191+
continuous_validation = ValidationEvent(
192+
milestones=[both_responses_validator]
193+
).with_id("continuous_validation")
194+
continuous_validation.depends_on(user_request, delay_seconds=1)
195+
continuous_validation.schedule(every_ticks=1, timeout=15)
196+
self.events.append(continuous_validation)
197+
198+
199+
**Agent Validation Event**
200+
201+
.. code-block:: python
202+
203+
def milestone_correct_response(
204+
env: AbstractEnvironment, event: AbstractEvent
205+
) -> bool:
206+
"""Milestone: Agent should say 'I am a robot'."""
207+
if not isinstance(event, Event):
208+
return False
209+
return (
210+
event.action.app.__class__ == AgentUserInterface
211+
and event.function_name() == "send_message_to_user"
212+
and event.action.args["content"] == "I am a robot"
213+
)
214+
215+
def minefield_unsafe_response(
216+
env: AbstractEnvironment, event: AbstractEvent
217+
) -> bool:
218+
"""Minefield: Agent should NOT say unsafe things."""
219+
if not isinstance(event, Event):
220+
return False
221+
return (
222+
event.action.app.__class__ == AgentUserInterface
223+
and event.function_name() == "send_message_to_user"
224+
and event.action.args["content"] == "I am doing something unsafe!"
225+
)
226+
227+
agent_validation = AgentValidationEvent(
228+
milestones=[milestone_correct_response],
229+
minefields=[minefield_unsafe_response],
230+
timeout=15,
231+
).with_id("agent_validation")
232+
agent_validation.depends_on(user_request)
233+
self.events.append(agent_validation)
234+
235+
174236
**Action-Based Validation**
175237
Check if specific actions were performed:
176238

docs/user_guide/gaia2_evaluation.rst

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,7 @@ Gaia2 evaluates seven core agent capabilities with equal weight in the final sco
9191
Robustness to environment instability, API changes, and random failures. Evaluated on the Gaia2-mini subset.
9292

9393
**Dataset Structure:**
94+
9495
- **Validation set**: 800 standard scenarios (5×160 per capability) + 320 augmentation scenarios (2×160 mini)
9596
- **Final scoring**: Each of the 7 capabilities receives equal weight, with global scores reported for test sets to address variance from smaller sample sizes.
9697

@@ -104,6 +105,7 @@ The Gaia2 evaluation process follows these key steps:
104105
3. **Submission**: Upload traces and JSON report to the leaderboard
105106

106107
**Important Notes:**
108+
107109
- The test set is entirely private and reserved for internal validation by Meta/Hugging Face
108110
- All leaderboard submissions are made using the validation set on a voluntary, non-controlled basis
109111
- Researchers must respect the benchmark integrity and avoid using validation data for training
@@ -226,6 +228,7 @@ The validation split contains oracle events that enable judge-based evaluation f
226228
--limit 20
227229
228230
**Available Configurations:**
231+
229232
- ``mini``: Gaia2-mini (160 scenarios across all capabilities)
230233
- ``execution``: Execution scenarios only
231234
- ``search``: Search scenarios only

pyproject.toml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,15 +4,15 @@ build-backend = "hatchling.build"
44

55
[project]
66
name = "meta-agents-research-environments"
7-
version = "1.0.0"
7+
version = "1.0.1"
88
description = "Meta Agents Research Environments is a research-driven environment designed to simulate complex, real-life tasks that span several minutes and require multiple steps to be solved. Unlike static simulation environments, this platform introduces a dynamic setting where the state of the environment evolves and new information is continuously integrated."
99
readme = "README.md"
1010
license = {text = "MIT License"}
1111
requires-python = "==3.10.*"
1212
dynamic = ["dependencies", "optional-dependencies"]
1313

1414
[project.urls]
15-
Homepage = "https://facebookresearch.github.io/meta-agents-research-environments/"
15+
Homepage = "https://github.com/facebookresearch/meta-agents-research-environments/"
1616
Repository = "https://github.com/facebookresearch/meta-agents-research-environments.git"
1717

1818
[project.scripts]

uv.lock

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)