Merge pull request #9 from constructive-io/feat/python-surface-templates

feat: python surface templates, gql and sql

Author: pyramation

README.md

@@ -7,28 +7,36 @@ The templates `fun init` scaffolds from: a **feature** in
 
 ```bash
 fun init billing --surface gql            # a GraphQL/HTTP feature — no database
-fun init billing --surface sql            # a db-connected feature, owning its SQL
+fun init billing --surface sql            # a db-connected feature — statements, not schema
 fun init billing --surface sql --kind sync --route /billing/export
+fun init billing --surface sql --lang python   # the same feature, in python
 fun init report --type python             # a platform handler
 ```
 
 ## Structure
 
 ```
-typescript/            handler/
-├── gql/               ├── node-multi-method/
-└── sql/               └── python/
+typescript/            python/             handler/
+├── gql/               ├── gql/            ├── node-multi-method/
+└── sql/               └── sql/            └── python/
 ```
 
 Two axes, and nothing else: the **surface** is what the function is connected to,
-the **language** is what you write it in. Python features wait on the python
-runtime mirror (constructive-planning#1455, phase 0.3) — until it lands, a
-feature is TypeScript and `handler/python` is how python code ships.
+the **language** is what you write it in. The four cells are the same feature
+shape twice, each at home in its own language — `handler.json` beside a
+`package.json` for node, beside a `requirements.txt` for python — over one
+runtime, mirrored in both (`functions/runtime` and `functions/runtime-py` in
+constructive-db), never two implementations free to disagree. `handler/*` is the
+separate thing: a *platform* handler in constructive-db's own tree.
+
+A **page** is a node image (`type: "node-page"`), so `--kind page` is a node
+feature; a python feature that serves a browser route puts those methods in a
+node manifest nested beside its own.
 
 | surface | what it reaches | database |
 |---|---|---|
 | `gql` | the tenant's API through `ctx.client`, its buckets, secrets and models | **none** — `ctx.db` is not on the context type |
-| `sql` | the same, plus a transaction through `ctx.db(fn)` | yes |
+| `sql` | the same, plus a transaction through `ctx.db(fn)` (`await ctx.db(fn)` in python) | yes |
 
 `ctx.db(fn)` runs the callback in one transaction that has assumed a
 low-privilege role and stamped the invocation's identity claims, so the tenant's
@@ -125,9 +133,29 @@ a container can touch, every route can. `methods[]` carries what makes a functio
 a *different* function — its task, its typed `inputs`/`outputs`/`props`.
 
 Dependencies are **not** in `handler.json`. A node feature declares them in the
-`package.json` beside it and a python handler in its `requirements.txt`, because
-those are the files node and pip actually read; image and system packages belong
-to the Dockerfile.
+`package.json` beside it and a python feature in its
+`handlers/requirements.txt`, because those are the files node and pip actually
+read; image and system packages belong to the Dockerfile.
+
+## What a python feature looks like
+
+The same four files, in python's shape. The image is `constructive_runtime` plus
+a FastAPI entry point that imports `handlers/handler.py` and serves every public
+coroutine in it as `POST /<name>` — so a method is an `async def`, discovered
+rather than registered, and `handlers/index.ts` has no python counterpart:
+
+```python
+async def export(params: Params, ctx: SqlContext) -> Result:
+    async def read(db: DbSession):
+        return await db.fetch("SELECT …")
+
+    return {"rows": await ctx.db(read)}
+```
+
+Its suite is TypeScript, because the platform it drives is: it registers the
+feature from `handler.json`, stages and starts the real python image
+(`startPythonImage`), and invokes it through the real queue. The first run builds
+the image's venv under `.image/`; later runs reuse it.
 
 ## Verifying a template
 

python/gql/.boilerplate.json

@@ -0,0 +1,25 @@
+{
+  "type": "module",
+  "requiresWorkspace": true,
+  "questions": [
+    {
+      "name": "____name____",
+      "message": "Feature name (directory, image and the task's category)",
+      "required": true
+    },
+    {
+      "name": "____method____",
+      "message": "Function name (the task is <feature>:<function>)",
+      "required": true
+    },
+    {
+      "name": "____version____",
+      "message": "Initial version",
+      "default": "0.0.1"
+    },
+    {
+      "name": "____description____",
+      "message": "Short description"
+    }
+  ]
+}

python/gql/.gitignore

@@ -0,0 +1,4 @@
+# The staged python image and its venv: rebuilt from the template, the runtime
+# and this feature's requirements whenever any of them change.
+.image/
+__pycache__/

python/gql/README.md

@@ -0,0 +1,108 @@
+# ____name____
+
+____description____
+
+```
+handlers/handler.json     what this feature is, and what it may reach
+handlers/handler.py       its functions — the image serves every public coroutine
+handlers/requirements.txt its pip dependencies, like any other python project
+__tests__/                the manifest, the image and the queue, end to end
+```
+
+## The surface
+
+This is a **gql** feature, written in python: it is served over HTTP/GraphQL and
+**has no database connection**. It reaches the tenant's data through the
+tenant's own API, which applies the caller's permissions, and its declared
+buckets, secrets and models through the rest of the context — never Postgres
+directly. The handler's `ctx` is typed as a protocol that has no `db`, so a
+query is a type error rather than a habit. A feature that reads the tenant's
+database directly is a **sql** feature
+(`fun init <name> --surface sql --lang python`).
+
+## The python image
+
+The image is `constructive_runtime` plus a FastAPI entry point that imports
+`handlers/handler.py` and serves **every public coroutine in it** as
+`POST /<name>` — the route the platform addresses `____name____:<name>` through.
+So a second function is a second `async def` and a second entry in `methods[]`;
+nothing else changes. A helper that must not become a route is either a plain
+`def` or lives in another module.
+
+The context is the same surface a TypeScript handler gets — `ctx.secrets`,
+`ctx.storage`, `ctx.agent`, `ctx.log`, `ctx.job` — because it is the same
+runtime, mirrored in python rather than reimplemented. The declared inputs are
+compiled once, at generation, into the JSON Schema both languages enforce, so a
+payload the node runtime would refuse is refused here too.
+
+The **kind** (`job`, `sync`, `page`) is not a different template, only a
+different way in: a job is enqueued and run by the worker, a sync is invoked
+through the gateway on the caller's connection. It is expressed in `handler.json`
+as `accessChannels` plus `route`, and `fun init --kind` fills it. A page is a
+node image (`type: "node-page"`), so a python feature serving one puts those
+methods in a node manifest beside this one.
+
+## What it may reach
+
+`handlers/handler.json` is this feature's declaration and the only place its
+identity and capabilities are written down — the platform reads it at deploy
+time, and the test reads the same file, so the two cannot drift.
+
+Anything undeclared is unreachable: `ctx.storage` and `ctx.secrets` raise on a
+key this file never declared rather than answering `None`. Declare what you use,
+as you write it:
+
+```json
+"requires": {
+  "buckets": ["exports"],
+  "secrets": [{ "name": "STRIPE_KEY", "required": true }],
+  "configs": [{ "name": "EXPORT_ROW_LIMIT", "required": false }],
+  "modules": ["notifications_module"],
+  "models": ["gpt-4o"]
+}
+```
+
+Always the **logical** key, never a physical name:
+`ctx.storage.write('exports', …)` resolves to this tenant's bucket per
+invocation, and `ctx.secrets.get('STRIPE_KEY')` reads from this tenant's own
+store. Secret *values* never travel in the manifest, the capability bundle, the
+payload, the logs, or the pod's environment.
+
+pip dependencies go in `handlers/requirements.txt`, where python already keeps
+them — the image installs it on top of the runtime's own. `handler.json` carries
+only what the platform reads, and system packages belong to the Dockerfile.
+
+## Running it
+
+```bash
+pgpm docker start --image docker.io/constructiveio/postgres-plus:18
+eval "$(pgpm env)"
+pnpm --filter "@constructive-functions/feature-____name____" test
+```
+
+The suite is TypeScript because the platform it drives is: it registers this
+feature from `handler.json`, stages and starts the real python image, and
+invokes it through the real queue. The first run builds the image's venv under
+`.image/` (git-ignored) and later runs reuse it, so only the first is slow.
+
+## Next
+
+The test above is the loop: it clones a seeded template database, needs no
+cluster, and is the only thing you need while writing the handler. When you want
+this feature on a real stack, from the root of the checkout it lives in:
+
+```bash
+pnpm fun up --k8s      # brings the platform up and registers every feature here
+```
+
+Registration reads `handlers/handler.json` — the same file the test reads — so a
+manifest-only change needs no rebuild:
+
+```bash
+pnpm fun register --apply     # write the declaration; --dry-run prints the SQL
+```
+
+A registration failure aborts the bring-up rather than being reported as
+skipped, which it once was: an unregistered method has no symptom of its own
+until something calls it and gets
+`No service URL for "____name____:____method____"`.

python/gql/__tests__/queue.test.ts

@@ -0,0 +1,85 @@
+import path from 'node:path';
+
+import type { FunctionsTestResult, RunningImage } from '@constructive-functions/test-utils';
+import {
+  addJob,
+  getConnections,
+  registerFeature,
+  resolveDatabaseId,
+  runQueuedJobs,
+  startPythonImage
+} from '@constructive-functions/test-utils';
+
+/**
+ * The platform test: this feature registered from the very file the platform
+ * reads (`handlers/handler.json`), served by the real python image, invoked
+ * through the real queue.
+ *
+ * A python feature has no `handlers/index.ts` to import — the image discovers
+ * its methods from `handler.py`, so nothing here reaches into the handler's
+ * code. What runs is `main.py` importing `handler.py` through
+ * `constructive_runtime`, with only the container removed; the first time it
+ * runs, the harness builds the image's venv, which is why the timeout is
+ * generous.
+ *
+ * This is what makes `handler.json` authoritative rather than decorative — a
+ * capability the manifest forgot to declare fails here, before deploy, and a
+ * task identifier that disagrees with `<category>:<name>` fails at registration.
+ */
+const featureDir = path.resolve(__dirname, '..');
+
+const IMAGE = 'features/____name____';
+const TASK = '____name____:____method____';
+
+let conn: FunctionsTestResult;
+// The worker claims jobs and pins a connection per dispatch, so it runs on the
+// harness-owned pool rather than the suite's transaction-bound client.
+let pool: ReturnType<FunctionsTestResult['getPool']>;
+let image: RunningImage;
+let databaseId: string;
+
+beforeAll(async () => {
+  conn = await getConnections();
+  pool = conn.getPool();
+  databaseId = await resolveDatabaseId(pool);
+
+  // No pool is handed to the image: this feature has no database connection,
+  // and one given here would be one `ctx.db` could use.
+  image = await startPythonImage({ name: IMAGE, featureDir });
+  await registerFeature(pool, databaseId, featureDir, { image: IMAGE });
+}, 600_000);
+
+// A job that failed is left queued for its retry — that is the queue working.
+// Each test starts from an empty one so a deliberate failure here is not the job
+// the next test's run picks up.
+afterEach(async () => {
+  await pool.query('DELETE FROM app_jobs.jobs WHERE database_id = $1', [databaseId]);
+});
+
+afterAll(async () => {
+  await image?.close();
+  await conn?.teardown();
+});
+
+describe('____name____:____method____ through the platform', () => {
+  it('runs the job the platform enqueues', async () => {
+    await addJob(pool, databaseId, TASK, { subject: 'a subject' }, { entity_type: 'platform' });
+
+    const { jobs, log } = await runQueuedJobs({ pool, databaseId, images: [image] });
+
+    expect(jobs).toHaveLength(1);
+    expect(log.entries).toEqual([
+      expect.objectContaining({ task_identifier: TASK, status: 'completed' })
+    ]);
+  });
+
+  // The declaration is the check: the handler asks nothing about its payload,
+  // because a payload the manifest does not allow never reaches it.
+  it('refuses a payload the manifest does not allow', async () => {
+    await addJob(pool, databaseId, TASK, {}, { entity_type: 'platform' });
+
+    await expect(runQueuedJobs({ pool, databaseId, images: [image] })).rejects.toThrow(
+      /subject is required/
+    );
+  });
+});

python/gql/handlers/handler.json

@@ -0,0 +1,32 @@
+{
+  "name": "____name____",
+  "version": "____version____",
+  "description": "____description____",
+  "type": "python",
+  "port": 8080,
+  "scope": "platform",
+  "image": "features/____name____",
+  "runtime": "http",
+  "accessChannels": [],
+  "requires": {
+    "buckets": [],
+    "modules": [],
+    "models": [],
+    "secrets": [],
+    "configs": []
+  },
+  "methods": [
+    {
+      "taskIdentifier": "____name____:____method____",
+      "description": "____description____",
+      "icon": "code",
+      "category": "____name____",
+      "inputs": [
+        { "name": "subject", "type": "string", "description": "What the call is about" }
+      ],
+      "outputs": [
+        { "name": "ok", "type": "boolean", "description": "Whether the call succeeded" }
+      ]
+    }
+  ]
+}

python/gql/handlers/handler.py

@@ -0,0 +1,62 @@
+"""
+____name____:____method____ — the gql surface, in python.
+
+The image's entry point imports this module and serves every public coroutine in
+it as `POST /<name>`, which is how the platform addresses `____name____:<name>`.
+A second function is a second coroutine here and a second entry in `methods[]`;
+nothing else changes. A helper that must not become a route is either a plain
+`def` or lives in another module.
+"""
+
+from typing import Any, Protocol, TypedDict
+
+
+class Params(TypedDict):
+    """
+    The inputs `handler.json` declares, as types.
+
+    `subject` is a required string there, so it is a `str` here: the runtime
+    compiles the declaration to JSON Schema and answers a payload that violates
+    it with a 400 before this coroutine is entered. That is why there is no
+    check for it below — a presence check here would be re-asking a question the
+    platform has already refused the request over, and an optional port is
+    `optional: true` in the manifest rather than an `if` in the body.
+    """
+
+    subject: str
+
+
+class Result(TypedDict):
+    ok: bool
+
+
+class GqlContext(Protocol):
+    """
+    What this function is handed as `ctx`: the platform's context, **without the
+    database**.
+
+    A GraphQL/HTTP-served function reaches the tenant's API and its declared
+    buckets, secrets and models, never Postgres directly. The runtime passes its
+    own `FunctionContext`, which carries a `db`; naming the surface here is what
+    keeps it out of this function — a type checker rejects `ctx.db`, exactly as
+    the node template's `Omit<FunctionContext, 'db'>` does. A feature that reads
+    the tenant's database is scaffolded with `--surface sql` instead.
+    """
+
+    job: dict
+    agent: Any
+    secrets: Any
+    storage: Any
+    log: Any
+    env: dict
+    capabilities: Any
+
+
+async def ____method____(params: Params, ctx: GqlContext) -> Result:
+    """
+    Raising is how this function reports failure — the platform records it and
+    retries. Returning an `{"ok": False}` of your own invention hides the
+    failure from both.
+    """
+    ctx.log.info("____name____:____method____", {"subject": params["subject"]})
+    return {"ok": True}