Improve documentation and add sample eval (#31)

* add sample eval, update missing list

* more doc tweaks

* cost warning

* clarify subnet

* add more security warning

Author: art-dsit

README.md

@@ -7,18 +7,18 @@ as sandboxes, running within AWS EC2.
 
 ## Installing
 
-Add this using [Poetry](https://python-poetry.org/)
+Add this using [uv](https://github.com/astral-sh/uv)
 
 ```
-poetry add git+ssh://git@github.com/UKGovernmentBEIS/inspect_ec2_sandbox.git
-```
-
-or in [uv](https://github.com/astral-sh/uv),
+uv add git+ssh://git@github.com/UKGovernmentBEIS/inspect_ec2_sandbox.git
+``` 
+or in [Poetry](https://python-poetry.org/),
 
 ```
-uv add git+ssh://git@github.com/UKGovernmentBEIS/inspect_ec2_sandbox.git
+poetry add git+ssh://git@github.com/UKGovernmentBEIS/inspect_ec2_sandbox.git
 ```
 
+
 ## AWS Infrastructure
 
 This plugin depends on certain infrastructure existing already. See the [infra docs](infra/README.md) for a
@@ -133,13 +133,31 @@ sandbox=SandboxEnvironmentSpec("ec2", Ec2SandboxEnvironmentConfig.from_settings(
 
 See [schema.py](src/ec2sandbox/schema.py) for details.
 
+## Compatibility with existing Inspect evals
+
+This sandbox provider is not [Dockerfile-compatible](https://inspect.aisi.org.uk/sandboxing.html#environment-binding).
+Hence, you will have to rebuild your evaluation envionment on top of an AWS virtual machine AMI.
+
+It is not a drop-in replacement; passing `--sandbox ec2` is unlikely to work for an existing eval.
+
+## Sample evaluation
+
+You can look at an existing [sample eval](src/ec2sandbox/examples/where_am_i.py) for how to get started.
+
+A more complex eval is [Sandbox Escape Bench](https://github.com/UKGovernmentBEIS/sandbox_escape_bench), where this EC2 sandbox is used as an outer sandbox and the agent is tasked with escaping an inner sandbox, generally Docker or Kubernetes.
+
+## Sandboxing limitations
+
+A sandbox is not a magic bullet for AI agent security. For more background, read AISI's [Inspect Sandboxing Toolkit](https://github.com/UKGovernmentBEIS/aisi-sandboxing).
+
+For the EC2 sandbox specifically, beware that the IAM role used by the sandbox EC2 instance (which is required by this provider for communication) can also be used by the agent.
+
+
 ## Tech Debt / Missing features
 
-- task_cleanup is not implemented; only the default sample_cleanup is, so if you Ctrl-C a run, you have to clean up with the CLI command
 - Move long-running AWS commands to a separate thread to avoid blocking Inspect's TUI
-- Integration testing of all features
-- Add more sample evals
 - better logging/tracing
+- Dockerfile-compatibility
 
 
 ## Developing

infra/README.md

@@ -3,11 +3,13 @@
 This folder contains a CDK project that allows you to deploy a basic VPC for 
 use by Inspect EC2 sandboxes.
 
+**WARNING**: This stack will result in ongoing AWS charges even when you have no sandboxes active: as of July 2026, around $60/month.
+
 ## Architecture
 
 It creates a VPC with three subnets, all in a single AZ.
 
-One of the subnets is completely isolated from the internet, the other is private with outgoing internet access.
+One of the subnets is completely isolated from the internet, the other is private with outgoing internet access. (The third is used only for the NAT gateway, not sandbox VMs.)
 
 It creates an S3 bucket which is needed for data communication with sandbox EC2 instances.
 

src/ec2sandbox/examples/__init__.py

@@ -0,0 +1 @@
+"""Runnable sample evals for the EC2 sandbox provider."""

src/ec2sandbox/examples/where_am_i.py

@@ -0,0 +1,31 @@
+"""Sample eval: have the agent work out which EC2 instance it is running on."""
+
+from inspect_ai import Task, task
+from inspect_ai.dataset import Sample
+from inspect_ai.scorer import match
+from inspect_ai.solver import basic_agent
+from inspect_ai.tool import bash
+from inspect_ai.util import SandboxEnvironmentSpec
+
+from ec2sandbox.schema import Ec2SandboxEnvironmentConfig
+
+INSTANCE_TYPE = "t3a.small"
+
+PROMPT = """\
+You are running inside a virtual machine on AWS. Work out what \
+type of virtual machine you are on, and submit its instance type identifier \
+(for example, "m5.xlarge")."""
+
+
+@task
+def where_am_i() -> Task:
+    """Ask the agent to discover and report its own EC2 instance type."""
+    return Task(
+        dataset=[Sample(input=PROMPT, target=INSTANCE_TYPE)],
+        solver=basic_agent(tools=[bash(timeout=60)], message_limit=20),
+        scorer=match(location="exact"),
+        sandbox=SandboxEnvironmentSpec(
+            "ec2",
+            Ec2SandboxEnvironmentConfig.from_settings(instance_type=INSTANCE_TYPE),
+        ),
+    )