Skip to content

Commit 552ea0c

Browse files
opencolinclaude
andcommitted
Add Prefect worker example
Shows Prefect 3.x flow runs executing in per-run Tenki microVMs via the prefect-tenki worker (installed from git at a pinned commit; not on PyPI yet). verify.py drives TenkiWorker.run() directly against live Tenki — mirroring upstream's live_worker_smoke.py — since a localhost Prefect server is unreachable from inside the guest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 1c86a86 commit 552ea0c

5 files changed

Lines changed: 175 additions & 0 deletions

File tree

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@ Give an agent a sandboxed place to run the code it writes.
6363
| [Covalent](examples/covalent-tenki/) | Each workflow task in its own microVM |
6464
| [AgentBox](examples/agentbox-tenki/) | Coding-agent boxes as Firecracker microVMs |
6565
| [ComputeSDK](examples/computesdk-tenki/) | Tenki as a provider for the unified sandbox interface |
66+
| [Prefect](examples/prefect-tenki/) | Each flow run in its own microVM via a Prefect 3.x worker |
6667

6768
### Agent platforms
6869

examples/prefect-tenki/README.md

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# Prefect flow runs on Tenki
2+
3+
Run [Prefect 3.x](https://docs.prefect.io) flow runs where **each flow run executes in its own disposable [Tenki](https://tenki.cloud) microVM** — a fresh Linux VM with real root, created when the run is scheduled and destroyed the moment it finishes. In Prefect 3.x, a *worker* polls a *work pool* and provisions infrastructure per flow run; [`prefect-tenki`](https://github.com/luxorlabs/tenki-prefect) adds a worker type `tenki` that makes that infrastructure a Tenki sandbox.
4+
5+
> **Not on PyPI yet.** `prefect-tenki` is pre-alpha (`0.1.0.dev0`) and installs from git — [`requirements.txt`](requirements.txt) pins a known-good commit.
6+
7+
## Setup (Python 3.10+)
8+
9+
```bash
10+
uv venv # or: python3.11 -m venv .venv
11+
uv pip install -r requirements.txt # prefect-tenki (from git) + prefect + tenki SDK
12+
export TENKI_API_KEY=tk_... # from the Tenki dashboard
13+
```
14+
15+
## Walkthrough
16+
17+
**1. Create a work pool.** Installing `prefect-tenki` registers the worker type via a `prefect.collections` entry point, so the CLI discovers it:
18+
19+
```bash
20+
prefect work-pool create --type tenki my-tenki-pool
21+
```
22+
23+
**2. Save credentials as a block.** The API key lives in a Prefect `TenkiCredentials` block, used only by the worker process to call the Tenki API — it is **never injected into the guest VM's environment**:
24+
25+
```python
26+
from prefect_tenki import TenkiCredentials
27+
28+
TenkiCredentials(api_key="tk_...").save("tenki-prod")
29+
```
30+
31+
**3. Set job variables** on the pool (UI or `prefect work-pool update`): `workspace_id`, sizing (`cpu_cores` default 2, `memory_mb` default 4096, optional `disk_size_gb`), and `image` *or* `snapshot_id`.
32+
33+
> **The image must be able to run Prefect.** Inside the sandbox the worker runs Prefect's prepared `prefect flow-run execute` command — the guest pulls your flow code itself. So the image/snapshot needs **Prefect + your flow's dependencies preinstalled**, and `allow_outbound: true` (the default) so it can **reach the Prefect API and your code source**. That also means the Prefect API must be reachable *from the microVM*: use Prefect Cloud or a server on a real address — `http://localhost:4200` only exists on your laptop.
34+
35+
**4. Start a worker** (any machine with the venv + credentials — your laptop works):
36+
37+
```bash
38+
prefect worker start --pool my-tenki-pool
39+
```
40+
41+
**5. Deploy and run:**
42+
43+
```bash
44+
python -c "
45+
from prefect import flow
46+
47+
@flow(log_prints=True)
48+
def hello(): print('hello from a Tenki microVM')
49+
50+
hello.from_source('https://github.com/you/your-repo', entrypoint='flows.py:hello') \
51+
.deploy(name='hello-tenki', work_pool_name='my-tenki-pool')
52+
"
53+
prefect deployment run 'hello/hello-tenki'
54+
```
55+
56+
Each run now gets its own microVM: the worker creates a sandbox (`create_timeout_seconds`, default 180 s), streams stdout/stderr into the flow-run logs, and closes it when the command exits.
57+
58+
## Lifecycle guarantees
59+
60+
- **Credential isolation** — the API key goes only into the worker's SDK client; the guest env gets Prefect's own variables, never your Tenki credentials.
61+
- **Cancellation-shielded cleanup** — sandbox create/close run inside shielded scopes, so a cancelled or crashing run can't leak a VM mid-request; teardown always runs.
62+
- **`kill_infrastructure`** — cancelling a flow run in Prefect terminates the sandbox by ID (the sandbox ID is the run's infrastructure PID).
63+
- **Server-side backstop**`max_duration_seconds` (default 3600) self-destructs the sandbox even if the worker dies. No leaked billing.
64+
65+
## Verify (fast smoke check)
66+
67+
```bash
68+
node verify.mjs # or: .venv/bin/python verify.py
69+
```
70+
71+
CI can't run a real flow: the guest must reach the Prefect API, and a localhost `prefect server` is unreachable from inside the microVM. So [`verify.py`](verify.py) does what the upstream repo's own live smoke test does — checks the worker type registers, then drives **`TenkiWorker.run()` directly** with a stand-in command: create a live microVM → run it → assert exit 0 → teardown. That exercises everything Tenki-specific in the worker; the Prefect-side orchestration is covered by Prefect itself and the plugin's CI in [luxorlabs/tenki-prefect](https://github.com/luxorlabs/tenki-prefect).
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
prefect-tenki @ git+https://github.com/luxorlabs/tenki-prefect.git@1778874be00ddd725137e4aa1fd23e5d8d744f76

examples/prefect-tenki/verify.mjs

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
/**
2+
* Shim so the cookbook's Node verify harness can run this Python example.
3+
* Runs verify.py with the example's local venv if present, else `python3`.
4+
* Setup (Python 3.10+): `uv venv && uv pip install -r requirements.txt` (see README).
5+
*/
6+
import { spawnSync } from "node:child_process";
7+
import { existsSync } from "node:fs";
8+
9+
const py = existsSync(".venv/bin/python") ? ".venv/bin/python" : process.env.PYTHON || "python3";
10+
const r = spawnSync(py, ["verify.py"], { stdio: "inherit" });
11+
process.exit(r.status ?? 1);

examples/prefect-tenki/verify.py

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
"""
2+
Smoke-verify the Prefect example without a Prefect server:
3+
1. `prefect-tenki` imports and registers the "tenki" worker type, and
4+
2. `TenkiWorker.run()` drives the real per-flow-run infrastructure lifecycle
5+
against live Tenki — create microVM → run command → teardown — exit 0.
6+
7+
We deliberately do NOT execute a full flow run: the guest must reach the
8+
Prefect API, and a localhost `prefect server` is unreachable from inside the
9+
microVM. Driving run() directly is the upstream repo's own smoke pattern
10+
(tests/live_worker_smoke.py) and exercises everything Tenki-specific.
11+
12+
Token/workspace from env (CI) or ~/.config/tenki/config.yaml (local `tenki login`).
13+
"""
14+
import logging
15+
import os
16+
import sys
17+
import tempfile
18+
19+
# Keep ~/.prefect untouched — prefect writes its profile/db under PREFECT_HOME.
20+
os.environ.setdefault("PREFECT_HOME", tempfile.mkdtemp(prefix="prefect-tenki-verify-"))
21+
22+
import anyio
23+
from tenki import AsyncClient
24+
25+
from prefect_tenki import TenkiCredentials, TenkiWorker, TenkiWorkerJobConfiguration
26+
27+
assert TenkiWorker.type == "tenki", TenkiWorker.type # the work-pool type name
28+
29+
30+
def cfg(key):
31+
try:
32+
with open(os.path.expanduser("~/.config/tenki/config.yaml")) as f:
33+
for line in f:
34+
if line.startswith(key + ":"):
35+
return line.split(":", 1)[1].strip()
36+
except Exception:
37+
pass
38+
return ""
39+
40+
41+
token = os.environ.get("TENKI_AUTH_TOKEN") or os.environ.get("TENKI_API_KEY") or cfg("auth_token")
42+
if not token:
43+
print("No token. Set TENKI_AUTH_TOKEN, or run `tenki login`.")
44+
sys.exit(1)
45+
46+
# SDK auth gap: a bare `tenki login` session token is sent as Bearer (rejected);
47+
# prefix it `cookie:` so it goes as a cookie. A `tk_` API key works as-is.
48+
if not token.startswith(("tk_", "ory_st_", "cookie:")):
49+
token = f"cookie:{token}"
50+
51+
52+
async def main():
53+
credentials = TenkiCredentials(api_key=token)
54+
55+
workspace_id = os.environ.get("TENKI_WORKSPACE_ID") or cfg("current_workspace_id")
56+
if not workspace_id:
57+
async with AsyncClient(**credentials.get_client_options()) as client:
58+
workspaces = list((await client.who_am_i()).workspaces)
59+
if len(workspaces) != 1:
60+
raise RuntimeError(f"{len(workspaces)} workspaces accessible; set TENKI_WORKSPACE_ID.")
61+
workspace_id = workspaces[0].id
62+
63+
# Same shape a `prefect worker start` builds per flow run, minus the
64+
# Prefect-prepared `prefect flow-run execute` command.
65+
configuration = TenkiWorkerJobConfiguration(
66+
name="prefect-tenki-verify",
67+
command="set -eu; test \"$(python3 -c 'print(6*7)')\" = \"42\"; echo ok",
68+
credentials=credentials,
69+
workspace_id=workspace_id,
70+
cpu_cores=1,
71+
memory_mb=1024,
72+
allow_inbound=False,
73+
allow_outbound=False,
74+
max_duration_seconds=120,
75+
create_timeout_seconds=180,
76+
command_timeout_seconds=60,
77+
stream_output=True,
78+
)
79+
worker = TenkiWorker(work_pool_name="prefect-tenki-verify")
80+
result = await worker.run(object(), configuration)
81+
if result.status_code != 0:
82+
raise AssertionError(f"exit {result.status_code} (sandbox {result.identifier})")
83+
print("✓ prefect-tenki: TenkiWorker.run() → live microVM → flow command exit 0 → teardown")
84+
85+
86+
logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
87+
try:
88+
anyio.run(main)
89+
except Exception as e: # noqa
90+
print(f"✗ {type(e).__name__}: {e}")
91+
sys.exit(1)

0 commit comments

Comments
 (0)