Skip to content

orders.md

Latest commit

 

History

History
425 lines (350 loc) · 23.2 KB

orders.md

File metadata and controls

425 lines (350 loc) · 23.2 KB
Raw
title Orders

Last verified against code: 2026-03-01

Summary

Orders are Gas City's derived mechanism (Layer 2-4, part of Formulas & Molecules) for scheduled and event-driven work dispatch without human intervention. Each order pairs a trigger condition (when to fire) with an action (a shell script or formula wisp), living as an order.toml file inside formula directories. The controller evaluates all non-manual triggers on every patrol tick and dispatches due orders -- exec orders run shell scripts directly with no LLM involvement, while formula orders instantiate wisps dispatched to agent pools.

Key Concepts

  • Order: A parsed definition from an order.toml file with a Name (derived from subdirectory name), a dispatch action (Formula or Exec, mutually exclusive), a Trigger type, trigger-specific parameters, and optional Pool routing. Defined in the Order struct at internal/orders/order.go.

  • Trigger: The trigger condition that controls when an order fires. Five types exist: cooldown (minimum interval since last run), cron (5-field schedule matching), condition (shell command exits 0), event (matching events after a cursor position), and manual (explicit invocation only, never auto-fires). See internal/orders/triggers.go.

  • Exec Order: An order whose action is a shell command (exec field) run directly by the controller. No LLM, no agent, no wisp. The script receives ORDER_DIR in its environment, set to the directory containing the order.toml file. Default timeout: 60 seconds.

  • Formula Order: An order whose action is a formula name (formula field). When the trigger opens, the controller calls MolCook to instantiate a wisp and labels it for pool dispatch. Default timeout: 30 seconds.

  • ScopedName: A rig-qualified key that creates unique identity for orders across rigs. City-level orders use the plain name (e.g., dolt-health). Rig-level orders append :rig:<rigName> (e.g., dolt-health:rig:demo-repo). ScopedName drives independent cooldown tracking, event cursors, and label scoping.

  • Formula Layer: A directory scanned for orders/*/order.toml. Layers are ordered lowest to highest priority; a higher-priority layer's order definition overrides a lower-priority one with the same subdirectory name (last-wins semantics).

  • Tracking Bead: A bead created synchronously before each dispatch goroutine launches, labeled order-run:<scopedName>. Serves dual purpose: prevents the cooldown trigger from re-firing on the next tick, and provides execution history for gc order history.

Architecture

The order subsystem spans two packages:

  • internal/orders/ -- parsing, validation, scanning, and trigger evaluation. Pure library code with no side effects beyond shell command execution for condition triggers.
  • cmd/gc/ -- controller-side dispatch (order_dispatch.go) and CLI commands (cmd_order.go). Wires the library into the controller loop and the gc order command tree.
                ┌──────────────────────────────────────────────────┐
                │           Controller Tick                        │
                │           cmd/gc/controller.go                   │
                │                                                  │
                │  ┌────────────────────────────────────────────┐  │
                │  │  orderDispatcher.dispatch(ctx, now)    │  │
                │  │  cmd/gc/order_dispatch.go              │  │
                │  │                                             │  │
                │  │  for each order:                       │  │
                │  │    ┌───────────────────────────────────┐    │  │
                │  │    │ CheckTrigger(a, now, lastRunFn,      │    │  │
                │  │    │          ep, cursorFn)             │    │  │
                │  │    │ internal/orders/triggers.go      │    │  │
                │  │    └───────────┬───────────────────────┘    │  │
                │  │                │                             │  │
                │  │        ┌───────▼────────┐                   │  │
                │  │        │ TriggerResult.Due? │                   │  │
                │  │        └──┬──────────┬──┘                   │  │
                │  │       no  │          │ yes                   │  │
                │  │       skip│          ▼                       │  │
                │  │           │  Create tracking bead (sync)    │  │
                │  │           │          │                       │  │
                │  │           │          ▼                       │  │
                │  │           │  go dispatchOne(a) ──────────┐  │  │
                │  │           │     ┌────────┬────────┐      │  │  │
                │  │           │     │ IsExec │Formula │      │  │  │
                │  │           │     ▼        ▼        │      │  │  │
                │  │           │  shell    MolCook     │      │  │  │
                │  │           │  script   + label     │      │  │  │
                │  │           │     │        │        │      │  │  │
                │  │           │     └────┬───┘        │      │  │  │
                │  │           │          ▼            │      │  │  │
                │  │           │   Record event       │      │  │  │
                │  │           │   (fired/completed/  │      │  │  │
                │  │           │    failed)            │      │  │  │
                │  │           │                       │      │  │  │
                │  └───────────┴───────────────────────┴──────┘  │
                └──────────────────────────────────────────────────┘

                ┌──────────────────────────────────────────────────┐
                │          Order Discovery (Startup)          │
                │                                                  │
                │  buildOrderDispatcher()                     │
                │    ├─ cityFormulaLayers(cfg)                     │
                │    ├─ orders.Scan(cityLayers)               │
                │    ├─ for each rig:                              │
                │    │   ├─ rigExclusiveLayers(rigLayers, city)    │
                │    │   ├─ orders.Scan(exclusive)            │
                │    │   └─ stamp Rig field on each order     │
                │    ├─ filter out manual-trigger orders          │
                │    └─ return memoryOrderDispatcher           │
                └──────────────────────────────────────────────────┘

Data Flow

Discovery (on controller start and config reload):

  1. buildOrderDispatcher() resolves city-level formula layers via cityFormulaLayers() and calls orders.Scan() to find city orders.
  2. For each rig, rigExclusiveLayers() strips the city prefix from the rig's formula layers to avoid double-scanning city orders. The remaining rig-exclusive layers are scanned separately.
  3. Rig orders get their Rig field stamped with the rig name.
  4. Manual-trigger orders are filtered out (they never auto-dispatch).
  5. If no auto-dispatchable orders remain, the dispatcher is nil (nil-guard pattern -- callers check before use).

Trigger evaluation and dispatch (on each controller tick):

  1. dispatch() iterates all non-manual orders.
  2. CheckTrigger() evaluates the trigger condition against current time, last-run history (from bead store), and event state (from event bus).
  3. For each due order, a tracking bead is created synchronously with label order-run:<scopedName>. This is critical: it prevents the cooldown trigger from re-firing on the next tick.
  4. A goroutine calls dispatchOne() with a context timeout derived from effectiveTimeout() (per-order timeout capped by global max_timeout).
  5. dispatchOne() records an order.fired event, then branches:
    • Exec: dispatchExec() runs the shell command via ExecRunner, labels the tracking bead with exec (or exec-failed), and records order.completed or order.failed.
    • Formula: dispatchWisp() calls instantiateWisp() (which delegates to store.MolCook()), labels the wisp root bead with order-run:<scopedName> and pool:<qualifiedPool>, and records order.completed or order.failed.

Scanning (orders.Scan()):

  1. For each formula layer (ordered lowest to highest priority), read <layer>/orders/*/order.toml.
  2. Parse each TOML file into an Order struct. Set Name from the subdirectory name, Source from the absolute file path.
  3. Higher-priority layers overwrite lower ones by name (map keyed on subdirectory name).
  4. Exclude disabled orders (enabled = false) and those in the skip list.
  5. Return the result slice preserving discovery order.

Key Types

  • Order (internal/orders/order.go): The parsed order definition. Fields: Name, Description, Formula, Exec, Trigger, Interval, Schedule, Check, On, Pool, Timeout, Enabled, Source, Rig.

  • TriggerResult (internal/orders/triggers.go): The outcome of a trigger check. Fields: Due (bool), Reason (human-readable), LastRun (time.Time).

  • orderDispatcher (cmd/gc/order_dispatch.go): Interface with a single method dispatch(ctx, cityPath, now). Production implementation is memoryOrderDispatcher.

  • memoryOrderDispatcher (cmd/gc/order_dispatch.go): Holds the scanned order list, bead store, events provider, command runner, exec runner, events recorder, stderr writer, and max timeout.

  • ExecRunner (cmd/gc/order_dispatch.go): Function type func(ctx, command, dir string, env []string) ([]byte, error) for running shell commands. Production implementation shellExecRunner uses os/exec.

Invariants

These properties must hold for the order subsystem to be correct. Violations indicate bugs.

  • Formula XOR Exec: Every order has exactly one of formula or exec set. Validate() rejects orders with both or neither.

  • Exec orders have no pool: An exec order runs a shell script directly on the controller. It has no agent pipeline and therefore no pool. Validate() rejects exec + pool combinations.

  • Trigger type requires matching parameters: A cooldown trigger requires interval, a cron trigger requires schedule, a condition trigger requires check, an event trigger requires on. Validate() enforces these per-trigger-type constraints.

  • Tracking beads are created before dispatch goroutines: The tracking bead (labeled order-run:<scopedName>) is created synchronously in the main dispatch loop. This prevents the cooldown trigger from re-firing on the next controller tick while the dispatch goroutine is still running.

  • ScopedName provides rig isolation: The same order name deployed to multiple rigs produces independent scoped names (e.g., dolt-health:rig:rig-a vs dolt-health:rig:rig-b). Cooldown tracking, event cursors, and history queries all use ScopedName. Firing one rig's order does not affect another rig's trigger evaluation.

  • Higher-priority layers override lower by name: When the same order subdirectory name exists in multiple formula layers, Scan() uses the definition from the highest-priority layer (last in the layers slice). The override is total (the entire TOML definition replaces the lower one).

  • Manual triggers never auto-fire: CheckTrigger() for a manual trigger always returns Due: false. Manual orders are filtered out of the dispatcher entirely during build. They can only be triggered via gc order run.

  • Disabled orders are excluded from scan results: Scan() filters out orders with enabled = false. They do not appear in any CLI command output or dispatch evaluation.

  • Cron trigger fires at most once per minute: After matching the 5-field schedule, checkCron() verifies the last run was not in the same truncated minute. This prevents duplicate fires within a single cron window.

  • Event trigger uses cursor-based deduplication: Event orders track the highest processed event sequence number via seq:<N> labels on order-run beads. Formula orders stamp the wisp root or failure tracking bead; exec orders stamp the tracking bead before the command runs. Subsequent trigger checks use AfterSeq filtering to avoid reprocessing already-handled events.

  • Dispatch goroutines are drained on controller exit: Each due order launches a goroutine whose completion is tracked by an in-flight counter and channel signal on the dispatcher. Controller shutdown and config reload call orderDispatcher.drain(ctx) with a bounded timeout so tracking bead outcomes and order.failed / order.completed events are persisted before the dispatcher is discarded. Reload retains any dispatcher that does not drain before its timeout and drains it again during controller shutdown. Failed orders emit order.failed events but do not retry; the tracking bead prevents re-fire within the same cooldown window.

  • No role names in Go code: The order subsystem operates on config-driven pool names and formula references. No line of Go references a specific role name.

Interactions

Depends on How
internal/config OrdersConfig for skip list and max timeout. FormulaLayers for formula directory resolution. City struct for config access.
internal/events Recorder for emitting order.fired, order.completed, order.failed events. Provider for event trigger queries (List with AfterSeq filtering).
internal/beads Store for creating tracking beads, querying last-run history (ListByLabel), and instantiating wisps (MolCook). CommandRunner for bd CLI invocation.
internal/fsys FS interface for filesystem abstraction in Scan() (enables fake filesystem in tests). OSFS for production.
Depended on by How
cmd/gc/controller.go The controller loop calls buildOrderDispatcher() on startup and config reload, then calls dispatch() on each tick.
cmd/gc/cmd_order.go CLI commands (gc order list/show/run/check/history) use orders.Scan() and orders.CheckTrigger() for user-facing operations.
Health Patrol (cmd/gc/) Order dispatch is one phase of the Health Patrol tick cycle, running after agent reconciliation and wisp GC.

Code Map

File Responsibility
internal/orders/order.go Order struct, Parse(), Validate(), IsEnabled(), IsExec(), TimeoutOrDefault(), ScopedName()
internal/orders/triggers.go TriggerResult, CheckTrigger(), checkCooldown(), checkCron(), checkCondition(), checkEvent(), cronFieldMatches(), MaxSeqFromLabels()
internal/orders/scanner.go Scan() -- discovers orders across formula layers with priority override
cmd/gc/order_dispatch.go orderDispatcher interface, memoryOrderDispatcher, buildOrderDispatcher(), dispatch(), dispatchOne(), dispatchExec(), dispatchWisp(), effectiveTimeout(), rigExclusiveLayers(), qualifyPool(), ExecRunner, shellExecRunner
cmd/gc/cmd_order.go CLI commands: gc order list, show, run, check, history. Helper functions: loadOrders(), loadAllOrders(), cityFormulaLayers(), findOrder(), orderLastRunFn(), bdCursorFunc()

Configuration

Orders are defined as order.toml files inside formula directories following the structure <formulaDir>/orders/<name>/order.toml. The [orders] section in city.toml controls global order behavior.

order.toml (per-order definition)

[order]
description = "Check database health"
formula = "mol-db-health"        # dispatch action (XOR with exec)
# exec = "scripts/check-db.sh"   # alternative: shell script dispatch
trigger = "cooldown"                # cooldown | cron | condition | event | manual
interval = "5m"                  # required for cooldown trigger
# schedule = "0 3 * * *"         # required for cron trigger (5-field)
# check = "test -f /tmp/flag"    # required for condition trigger
# on = "bead.closed"             # required for event trigger
pool = "worker"                  # target pool for formula dispatch (optional)
timeout = "90s"                  # per-order timeout (optional)
enabled = true                   # default: true

city.toml (global settings)

[orders]
skip = ["noisy-order"]      # order names to exclude from scanning
max_timeout = "120s"             # hard cap on per-order timeout (default: uncapped)

Order layering (override priority, lowest to highest)

The formula layer order determines which order.toml wins when the same order name exists in multiple layers:

  1. City pack formulas -- from pack referenced in city.toml
  2. City local formulas -- from [formulas] section or .gc/formulas/
  3. Rig pack formulas -- from pack applied to a specific rig
  4. Rig local formulas -- from rig's formulas_dir

A higher-numbered layer completely replaces a lower-numbered layer's definition for the same order name. This enables packs to define defaults that operators override locally.

Rig-scoped orders

When a rig has rig-exclusive formula layers (layers beyond the city prefix), orders found in those layers are stamped with the rig name. This produces independent scoped tracking:

  • Same order deployed to rigs rig-a and rig-b tracks independently as db-health:rig:rig-a and db-health:rig:rig-b.
  • Pool names are auto-qualified: pool = "worker" in rig demo-repo becomes pool:demo-repo/worker on the wisp label. Already-qualified names (containing /) are left unchanged.

Testing

The order subsystem has comprehensive unit tests across three test files in the library and two in the CLI:

Test file Coverage
internal/orders/automation_test.go Parse (formula, exec, event orders), Validate (all trigger types, mutual exclusion, missing fields, timeout validation), IsEnabled default/explicit, IsExec, TimeoutOrDefault (defaults and custom), ScopedName (city and rig)
internal/orders/triggers_test.go CheckTrigger for all five trigger types: cooldown (never run, due, not due), cron (matched, not matched, already run this minute), condition (pass, fail), event (due, with cursor, cursor past all, not due, nil provider), rig-scoped triggers (cooldown, cron, event use ScopedName), MaxSeqFromLabels (various label configurations)
internal/orders/scanner_test.go Scan (basic discovery, empty layers, layer override priority, skip list, disabled filtering, source path recording)
cmd/gc/order_dispatch_test.go Dispatcher nil-guard (no orders, manual-only), cooldown dispatch (due, not due, multiple), exec dispatch (due, failure, cooldown, ORDER_DIR env, timeout), rig-scoped dispatch (rig stamping, independent cooldown, qualified pool), rigExclusiveLayers, qualifyPool, effectiveTimeout (default, custom, capped)
cmd/gc/cmd_order_test.go CLI commands: list (empty, with data, exec type), show (found, not found), check (due, not due), history, findOrder

All tests use in-memory fakes (fsys.NewFake(), beads.NewMemStore(), stubbed ExecRunner, memRecorder) with no external infrastructure dependencies. Condition trigger tests use real sh -c true and sh -c false commands. See TESTING.md for the overall testing philosophy and tier boundaries.

Known Limitations

  • No retry on dispatch failure: Failed orders emit events but are not retried. The tracking bead prevents re-fire within the same cooldown window, so a failed order must wait for the next trigger opening.

  • Cron granularity is minutes: The cron trigger operates at minute-level granularity with simple field matching (*, exact integer, comma-separated values). It does not support ranges (1-5), steps (*/5), or sub-minute scheduling.

  • Condition trigger blocks the dispatch loop: checkCondition() runs sh -c <check> synchronously during trigger evaluation. A slow check command blocks evaluation of subsequent orders on that tick.

  • Event trigger cursor is per-run, not per-dispatch: The cursor position is computed from seq:<N> labels on existing order-run beads via MaxSeqFromLabels(). The controller and gc order run fail closed when the current event head cannot be read. For side-effecting exec orders, the cursor is persisted before the command runs so a crash after execution does not replay the same event. Trade-off: a controller crash after the cursor stamp and before exec start drops that event for idempotent exec orders; for non-idempotent exec orders this is the safer failure mode.

  • No hot-add of orders: Order discovery runs on controller start and config reload (via fsnotify). Adding a new order.toml file requires the config directory watcher to trigger a reload; adding a new formula layer directory requires a city.toml change.

  • Fire-and-forget goroutines: Dispatch goroutines are not tracked by the controller. On shutdown, in-flight dispatches may be interrupted mid-execution if the context is canceled.

See Also