Add ephemeral EC2 Harbor backend

[andre-fu]

What changed

• Adds opt-in environment=ec2 execution with one public-IP, key-only SSH EC2 instance per Harbor trial.
• Keeps Daytona as the automatic CPU default; EC2 rejects accelerators, attach/retain mode, private networking, and platform-setting overrides.
• Gives trial workers dedicated EC2 control and SSH secrets, gives reconciliation only the control secret, and exposes neither secret to the API or dispatcher.
• Materializes AWS credentials and the SSH key in mode-0600 temporary files for the duration of the owning operation, then removes them.
• Tags instances and root volumes with protected account/deployment/trial/job/session/task ownership metadata.
• Persists a structured ec2://<account>/<region>/<instance> handle immediately after launch and ownership-checks every teardown.
• Terminates instances after normal completion, cancellation, stale-heartbeat cleanup, and scheduled orphan reconciliation.
• Adds a 30-minute orphan grace period and a 14-hour hard maximum age.
• Supports both the pinned in-process Harbor runtime and override-Harbor child runtime with the same resolved configuration and tags.
• Installs Harbor's EC2 dependencies and OpenSSH in standalone and Modal worker images.
• Documents IAM, networking, Modal secret, CLI, rollout, and canary requirements.

No ECS cluster, Auto Scaling group, database migration, or AWS infrastructure provisioning is introduced.

Why

Harbor needs a complete Docker/Compose or native k3s/Helm host. One disposable EC2 VM per trial provides that isolation while Modal remains only the worker dispatcher.

Safety and cleanup

• EC2 credentials are reference-counted behind one backend-owned lease so concurrent in-process runs cannot delete each other's temporary files.
• API cancellation delegates to a dedicated control-secret-bearing Modal function; that function resolves the backend directly and cannot recurse through the API delegate.
• Reconciliation snapshots AWS before opening the shared DB transaction and performs termination only after commit.
• Discovery and teardown both verify protected account/deployment ownership tags.
• AWS and teardown failures remain visible in structured logs and do not silently skip unrelated cleanup phases.

Validation

• 257 passed: EC2 backend, credentials, routing, policy, Harbor lifecycle patch, normal/override parity, ephemeral bridge, cleanup, and orphan decisions.
• 25 passed: hosted policy, Modal secret topology, and delegated teardown.
• Black 24.10.0 clean on all touched Python files.
• Ruff 0.8.4 F401,F841 clean and git diff --check clean.
• Existing PR checks and preview were green before the cleanup commit; fresh checks are running for the new head.

AWS canary

The SRE-world distractor-volume oracle task completed end-to-end on the pinned default Harbor runtime:

• instance type: m7i-flex.2xlarge
• native k3s/Helm environment
• reward: 1.0
• results uploaded successfully
• instance terminated and root volume count returned to zero
• cold duration: approximately 14m23s

Performance follow-up

The measured 8m49s environment_setup aggregates EC2 health waits, SSH/cloud-init, tool bootstrap, k3s startup, image pulls, Helm install, and pod readiness. Harbor does not yet emit subphase timings, so this aggregate does not establish which step dominates. Daytona uses the same serial k3s/Helm lifecycle after provisioning a tools-equipped sandbox; the EC2/Daytona delta therefore needs a same-task instrumented comparison. A prior 33 GB workload-image AMI was measurably slower because new EBS volumes still lazily initialized cached snapshot blocks.

Recommended follow-up order:

1. Fast-path teardown for owned disposable EC2 instances instead of synchronously uninstalling Helm/k3s and waiting for the terminated state.
2. Add structured subphase timings and compare the identical task on Daytona and EC2.
3. If the measurement supports it, start SSH probing after instance_running and canary bounded parallel kubelet image pulls.
4. Bake a small tools-only AMI containing k3s, kubectl, and Helm; do not bake the SRE workload image store unless Fast Snapshot Restore or explicit volume initialization is justified.

Daytona remains the default CPU backend after merge. EC2 is used only when explicitly selected.

---

Note

High Risk
Touches AWS instance launch/terminate, credential isolation, and cancellation paths; misconfiguration or tag/ledger mismatches could leak instances or refuse safe teardown, though the design is conservative and heavily tested.

Overview
Introduces an opt-in CPU Harbor backend (--env ec2) that runs one ephemeral public-IP EC2 instance per trial. Daytona remains the hosted CPU default; EC2 is registered only when ODDISH_EC2_ENABLED is set and is not auto-selected.

Modal deploy & secrets: Production, staging, and PR preview workflows turn on EC2 with platform launch coordinates; AWS control creds and the SSH key live in separate Modal secrets baked into an immutable deploy plan. API and dispatcher containers never receive those secrets—only ec2_trial workers get both, reconciler/teardown get control only. API cancellation calls a dedicated teardown_ec2_sandbox Modal function via a provider teardown delegate.

Workers & dispatch: worker_jobs gains an execution_lane (default vs ec2_trial) in the claim index and spawn routing. A dedicated process_single_ec2_trial_job path acquires sandbox_capacity_leases against ODDISH_EC2_MAX_CONCURRENT_INSTANCES before claiming queue slots. Dispatch units are now (queue_key, harbor_variant_id, execution_lane) across Docker/K8s/in-process backends.

Lifecycle & safety: New sandbox_runs ledger (per attempt, launch token, tagged ownership) and DB trigger to request termination when jobs leave RUNNING. Ec2Backend materializes mode-0600 AWS profile/SSH files, merges protected Harbor kwargs/tags, and refuses teardown without a matching ledger + tag tuple. Orphan reconciliation uses a pure policy (30m grace, 14h hard max).

Other: Pins Harbor to rev a7caa58 (EC2 support), adds boto3/openssh-client to worker images, ODDISH_EC2_* settings validation, cloud policy tests, and docs/CLI for explicit EC2 runs.

^{Reviewed by Cursor Bugbot for commit b16e479. Bugbot is set up for automated code reviews on this repo. Configure here.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
oddish-app	Ready	Preview	Aug 11, 2026 7:18am

[github-actions]

Oddish preview

Commit: b16e4792703867d86c7ca072c298ce2910061afa

Surface	Link	Target
Frontend	https://pr-1131.oddish.app	Vercel preview for b16e479
Backend	oddish-pr-1131	oddish-pr-1131
Database	project fwexqrgwlpuweknkmaqh	project fwexqrgwlpuweknkmaqh

Vercel deployment URL: https://oddish-gyij2rojz.oddish.app

Plan:

• Frontend deploy: true
• Backend deploy: true
• Migrations: false

This comment is updated by the PR Preview workflow.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 5a5620f. Configure here.}