Merge pull request #33 from eduralph/feat/in-driver-lane-concurrency

feat(driver): in-driver lane concurrency (worker pool + $PDCA_LANE)

Author: eduralph

README.md

@@ -80,6 +80,10 @@ leaves), so `init-issue` → `run` → `signoff` works before you wire anything
   both the driver and CI via one `pdca gates` command (stub fallback until filled).
 - **Batch fan-out + sign-off queue** — `pdca batch` over N issues, `pdca queue`
   cheap-first burn-down.
+- **In-driver lane concurrency** — `[driver].lanes = N` (`PDCA_LANES` /
+  `pdca flow|batch --lanes N`) fans the unattended Do + Check band across N workers in
+  one workspace; each gate sees its worker slot as `$PDCA_LANE` to keep checkouts /
+  runners lane-private (doc 09).
 - **Mechanical STOP discipline** — `.claude/agents/builder.md` + a PreToolUse hook
   block the builder from marking a PR ready/merging; `reviewer.md` has execute-only
   scope; the decorrelated reviewer path is cross-vendor Codex via `AGENTS.md`.

template/PCDA/quality-cycle/03-cycle-automation.md

@@ -12,7 +12,7 @@ status: active
 
 > **Continuous-flow extension.** Beyond the unattended `pdca run`, the driver can run the whole cycle as one continuous, Claude-driven flow — `pdca flow <id> [--from-csv …] [--act]` (or batch: `pdca flow --from-csv …`, one Plan session → several issues). This lifts the model from **two** leaves to **five**, *without* moving any control flow into a model: the planner (Plan — interactive, turns the human's documents into `brief.md`) and the sign-off and act leaves (interactive) instrument the human steps the principle above keeps human; Do (builder) and Check's reviewer stay **headless**. The state transitions, the gates, and the **C6 accept-guard remain deterministic code** — a leaf only fills an artifact. Leaves are configured in `pdca.toml` (`[leaves.*]`: `mode = stub|command`, `interactive`); set `PDCA_LEAVES_MODE=stub` to force the offline placeholders (CI / `make`).
 
-> **Parallel-lanes extension.** Because the bundle is the unit of isolation, several cycles can run **concurrently** for throughput — see [09 - Parallel Lanes](09-parallel-lanes.md). The key discipline: mechanical isolation (a private working tree per lane) makes concurrent *execution* safe, but correctness *across* the parallel results is a separate problem — handled by **lane planning** (group same-area issues into one lane) and the **merge re-gate** (`gates.run_working_tree` over the merged tree + the draft PR), never by isolation alone. Parallelism stays in the unattended Do + Check band; the human touch points remain serial.
+> **Parallel-lanes extension.** Because the bundle is the unit of isolation, several cycles can run **concurrently** for throughput — see [09 - Parallel Lanes](09-parallel-lanes.md). The key discipline: mechanical isolation (a private working tree per lane) makes concurrent *execution* safe, but correctness *across* the parallel results is a separate problem — handled by **lane planning** (group same-area issues into one lane) and the **merge re-gate** (`gates.run_working_tree` over the merged tree + the draft PR), never by isolation alone. Parallelism stays in the unattended Do + Check band; the human touch points remain serial. Two realizations: N separate workspaces (zero machinery), or the **in-driver worker pool** — `[driver].lanes = N` (`PDCA_LANES` / `--lanes N`) fans the Do + Check band across N workers in one workspace, each exposing its lane slot to gates as `$PDCA_LANE`.
 
 > **Maturity legend** — every major mechanism in this doc is tagged: **[built]** = ships in this template and runs today; **[partial]** = ships but needs per-project wiring (cells in [04 - Validation Tooling](04-validation-tooling.md) §Status today have the breakdown); **[project-provided]** = not shipped by the template — each project supplies it because it is tracker- or repo-specific. The driver (`pdca run` / `batch` / `queue` / `gates` / `act-index` / `act-log`, in `src/pdca_harness/`), the deterministic gate runner, the headless reviewer, the sign-off queue, and the Act-log tooling are all **[built]**. What each project still supplies: the **Plan-draft scaffolding** — a tracker scraper + handoff generator (the per-repo specification's item-9 tooling, **[project-provided]**); the **real gate check rows** (`pdca.toml`; an all-PASS stub fallback ships, **[partial]**); and the **real leaf commands** (the leaves run as offline stubs until a model is wired, **[partial]**).
 

template/PCDA/quality-cycle/08-glossary.md

@@ -206,10 +206,13 @@ status: active
   or long gate prints so a silent job doesn't look hung.
 - **Env overrides** — `PDCA_LEAVES_MODE=stub` (force all leaves offline) · `PDCA_GATES_MODE=stub`
   (stub the gates) · `PDCA_BUNDLE_ROOT` (redirect bundles, so a rehearsal can't collide
-  with the real `results/`).
-- **Lane** — an isolated execution context running cycles concurrently with other lanes
-  (an independent copy / `git worktree` of the workspace, own `results/` + own checkout).
-  Several lanes give throughput; the bundle is still the unit of isolation ([09](09-parallel-lanes.md)).
+  with the real `results/`) · `PDCA_LANES=N` (in-driver lane-pool size; overrides
+  `[driver].lanes`, [09](09-parallel-lanes.md)).
+- **Lane** — an isolated execution context running cycles concurrently with other lanes,
+  realized either as an independent copy / `git worktree` of the workspace (own `results/`
+  + own checkout) **or** as a worker slot in the in-driver pool (`[driver].lanes`), where a
+  gate scopes its checkout / runner by `$PDCA_LANE`. Several lanes give throughput; the
+  bundle is still the unit of isolation ([09](09-parallel-lanes.md)).
 - **Lane planning** — assigning issues to lanes by **code locality**: same-area fixes to
   one lane (serial), parallel only across disjoint areas. Partition by *what changes*, not
   by id; a Plan-beat judgment that prevents integration conflicts up front ([09](09-parallel-lanes.md)).

template/PCDA/quality-cycle/09-parallel-lanes.md

@@ -10,7 +10,7 @@ status: active
 
 > One level beside [03 - Cycle Automation](03-cycle-automation.md). How to run **several PDCA cycles at once** for throughput without the concurrent runs corrupting one another. Core principle: **the bundle is already the unit of isolation, so concurrent execution is safe once every *shared mutable resource* a cycle touches outside its bundle is made private to a lane — and correctness *across* the parallel results is a separate problem, solved by planning and the merge re-gate, never by isolation alone.** Living document.
 
-> **Maturity legend** (as in [03 - Cycle Automation](03-cycle-automation.md)): **[built]** ships in this template; **[project-provided]** is supplied per project because it is repo- or runner-specific. The integration primitives this doc relies on — per-issue bundles, the single-sourced gates (`pdca gates` over a bundle *and* over the working tree), and the publisher's draft PR — are **[built]**. The lane *mechanism* (how a project gives each lane its own working tree / checkout, and uniquely-named runner artifacts) is **[project-provided]**, because *what* must be isolated depends on what the project's gates and builder touch.
+> **Maturity legend** (as in [03 - Cycle Automation](03-cycle-automation.md)): **[built]** ships in this template; **[project-provided]** is supplied per project because it is repo- or runner-specific. The integration primitives this doc relies on — per-issue bundles, the single-sourced gates (`pdca gates` over a bundle *and* over the working tree), and the publisher's draft PR — are **[built]**. The **in-driver worker pool** (`[driver].lanes`, running the unattended Do+Check band concurrently and exposing each worker's lane slot to gates as `$PDCA_LANE`) is now **[built]** too. What remains **[project-provided]** is the lane *isolation itself* — deriving each lane's working tree / checkout, container names, ports, and scratch dirs *from* `$PDCA_LANE` inside the project's gate commands — because *what* must be isolated depends on what the project's gates and builder touch.
 
 ## A lane
 
@@ -74,7 +74,11 @@ So the shape is: **Plan (serial) → Do + Check fan out across lanes → sign-of
 The fan-out can be realized two ways, and they trade off cleanly:
 
 - **N separate workspaces** (the [project-provided] model above) — each lane is an independent, *serial* driver run in its own `$WORKSPACE`. This needs **no harness change**: the driver is serial and keeps no state outside its workspace, so N concurrent runs can't tangle at the harness level — all isolation is the filesystem boundary. The cost is full copies (disk) and a per-lane sign-off queue (the human attends each in turn).
-- **An in-driver worker pool** (one workspace, the driver running bundles concurrently) — lighter on disk and allows a single batched sign-off, but it is a **future enhancement**, not built: it requires the driver to run the unattended band concurrently *and* the harness to become lane-aware — making the **target checkout** (gates apply/revert in place) and the **runner artifacts** lane-scoped, which today are addressed by fixed derivation. That lane-awareness is the hard, non-local part.
+- **An in-driver worker pool** (one workspace, the driver running bundles concurrently) — lighter on disk (only N lane-scoped checkouts, reused across all bundles) and a single batched sign-off. This is now **[built]**: set `[driver].lanes = N` in `pdca.toml` (or `PDCA_LANES=N` / `--lanes N` for one run). The driver runs the unattended **Do + Check** band across a pool of N workers; **Plan, sign-off, publish, and Act stay serial** (the human band — and an `iterate-plan` re-open is re-planned in a serial pre-pass, never in the pool). Each worker is pinned to a fixed lane slot `0..N-1` for its lifetime and exposes it to every gate command as **`$PDCA_LANE`**.
+
+  The harness owns the *concurrency and the lane id*; it does **not** own the project's checkout or runner, so the actual isolation stays **[project-provided]**: a gate that applies/reverts a target checkout, or starts a container / binds a port / writes a scratch dir, must name that resource by `$PDCA_LANE` (e.g. a `repo-lane$PDCA_LANE` checkout, `--name app-l$PDCA_LANE`, `port = 8000 + $PDCA_LANE`) — exactly the "name runner artifacts uniquely per lane" rule above, now keyed off a harness-supplied slot. Because a worker reuses its slot across the bundles it pulls, only N copies are ever needed, not one per bundle. (Publish is unaffected — it runs in the serial join, so its checkout is never contended and needs no lane scoping.)
+
+  **Combine + re-gate** stays external and composes from existing [built] pieces: drive each lane's bundles into a per-lane bundle root if you want them separated (`PDCA_BUNDLE_ROOT`), then an external script merges the accepted lane branches and runs the repo-scoped merge re-gate (`pdca gates --working-tree`) over the combined tree (§Integration validation). The harness ships no lane-merge orchestrator — that's a project script over these primitives.
 
 Start with separate workspaces (it works today with the generic mechanics above); reach for the in-driver pool only when per-lane disk or sign-off ergonomics actually bite.
 

template/pdca.toml.jinja

@@ -13,6 +13,22 @@ bundle_root = "{{ bundle_root }}"
 process_dir = "{{ process_dir }}"
 templates_dir = "templates"
 
+# In-driver lane concurrency (docs 09 §parallel lanes). `lanes` is the worker-pool
+# size for the UNATTENDED Do+Check band only — Plan, sign-off, publish, and Act stay
+# serial. 1 (the default) is the strictly-serial driver. Override for a single run
+# with `PDCA_LANES=N` or `--lanes N` on `pdca flow` / `pdca batch`.
+#
+# A pool > 1 runs N bundles concurrently in ONE workspace. Each worker is pinned to a
+# fixed lane slot 0..N-1 and exposes it to every gate command as `$PDCA_LANE`. A gate
+# that backs onto a shared mutable resource (a target checkout it applies/reverts, a
+# container, a port, a scratch dir) MUST name that resource by `$PDCA_LANE` so two
+# lanes can't collide — e.g. `--name app-l$PDCA_LANE`, a `repo-lane$PDCA_LANE`
+# checkout. Bundles in the same lane reuse that lane's resources serially, so you need
+# only N copies, not one per bundle. Leave at 1 unless per-lane disk / sign-off
+# ergonomics actually bite (then prefer this over N separate workspaces).
+[driver]
+lanes = 1
+
 [tracker]
 system = "{{ tracker_system }}"
 url = "{{ tracker_url }}"

template/src/pdca_harness/cli.py

@@ -46,6 +46,7 @@ def main(argv: list[str] | None = None) -> int:
     p_flow.add_argument("--no-publish", action="store_true", help="don't open the draft PR after an accept")
     p_flow.add_argument("--act", action="store_true", help="run the Act leaf after a COMPLETE sign-off")
     p_flow.add_argument("--by", default="", help="who signed off (recorded in §9)")
+    p_flow.add_argument("--lanes", type=int, help="unattended Do+Check worker-pool size (docs 09; overrides [driver].lanes / PDCA_LANES)")
 
     p_status = sub.add_parser("status", help="list bundle states (cheap-first queue)")
     p_status.add_argument("issue_id", nargs="?")
@@ -55,6 +56,7 @@ def main(argv: list[str] | None = None) -> int:
     p_batch.add_argument("--from-briefs", type=Path, help="init missing bundles from DIR/<id>.md")
     p_batch.add_argument("--no-act", action="store_true", help="stop after sign-off; skip the end-of-batch Act")
     p_batch.add_argument("--by", default="", help="who signed off (recorded in §9)")
+    p_batch.add_argument("--lanes", type=int, help="unattended Do+Check worker-pool size (docs 09; overrides [driver].lanes / PDCA_LANES)")
 
     sub.add_parser("queue", help="the cheap-first sign-off burn-down (AWAITING_SIGNOFF)")
 
@@ -155,6 +157,8 @@ def _flow(cfg: Config, args: argparse.Namespace) -> int:
     a batch Plan session may brief several issues, which are then all built
     unattended and signed off cheap-first via the queue.
     """
+    if getattr(args, "lanes", None) is not None:
+        cfg.lanes = max(1, args.lanes)
     if args.issue_id:
         d = cfg.bundle(args.issue_id)
         if d.exists() and state.state(d) == state.COMPLETE:
@@ -214,6 +218,8 @@ def _batch(cfg: Config, args: argparse.Namespace) -> int:
     any missing bundle from DIR/<id>.md first. Resumable — already-COMPLETE ids are
     skipped, so re-running picks up whatever is still in flight.
     """
+    if getattr(args, "lanes", None) is not None:
+        cfg.lanes = max(1, args.lanes)
     # Seed any missing bundles from --from-briefs; sign-off and Act stay human.
     for issue_id in args.issue_ids:
         d = cfg.bundle(issue_id)

template/src/pdca_harness/config.py

@@ -74,6 +74,10 @@ class Config:
     gate_target_default: str = ""
     gate_target_match: dict[str, str] = field(default_factory=dict)
     gate_target_flags: dict[str, dict[str, str]] = field(default_factory=dict)
+    # In-driver lane concurrency (docs 09): the worker-pool size for the unattended
+    # Do+Check band. ``1`` (the default) keeps the driver strictly serial. ``[driver].lanes``
+    # in pdca.toml; ``PDCA_LANES`` overrides for a single run (like ``PDCA_BUNDLE_ROOT``).
+    lanes: int = 1
 
     def bundle(self, issue_id: str) -> Path:
         """The per-cycle bundle directory for an issue id."""
@@ -124,6 +128,13 @@ def leaf(name: str) -> LeafConfig:
             env_root = Path(os.environ["PDCA_BUNDLE_ROOT"])
             bundle_root = env_root if env_root.is_absolute() else root / env_root
 
+        # In-driver lane pool size. PDCA_LANES overrides [driver].lanes for one run
+        # (e.g. to rehearse parallelism without editing pdca.toml). Floor of 1 = serial.
+        lanes = int(data.get("driver", {}).get("lanes", 1))
+        if os.environ.get("PDCA_LANES"):
+            lanes = int(os.environ["PDCA_LANES"])
+        lanes = max(1, lanes)
+
         return cls(
             root=root,
             bundle_root=bundle_root,
@@ -149,6 +160,7 @@ def leaf(name: str) -> LeafConfig:
             act=leaf("act"),
             author=data.get("project", {}).get("author", ""),
             gates_checks=gates_checks,
+            lanes=lanes,
         )
 
 

template/src/pdca_harness/flow.py

@@ -21,9 +21,11 @@
 
 import datetime
 import sys
+import threading
+from collections import deque
 from pathlib import Path
 
-from . import driver, leaves, publish, queue, signoff, state
+from . import driver, lane, leaves, publish, queue, signoff, state
 from .config import Config
 
 
@@ -166,6 +168,53 @@ def flow(
     return final
 
 
+# ----------------------------------------------------------------------------
+# The unattended band: advance every bundle through Do + Check (docs 09). Serial by
+# default; a worker pool of cfg.lanes lanes when configured (PDCA_LANES / [driver].lanes).
+# ----------------------------------------------------------------------------
+def _build_all(cfg: Config, bundles: list[Path]) -> None:
+    """Drive each bundle through the unattended Do+Check band to AWAITING_SIGNOFF / COMPLETE.
+
+    ``cfg.lanes <= 1`` keeps the original strictly-serial loop (Plan-if-unplanned then
+    drive, per bundle). With ``cfg.lanes > 1`` the *drive* fans out across a worker pool:
+    a serial Plan pre-pass runs first (an ``iterate-plan`` may have re-opened a bundle to
+    UNPLANNED, and the Plan leaf is **interactive** — it must never enter the pool), then
+    ``min(lanes, len(bundles))`` worker threads, each pinned to a fixed lane slot for its
+    lifetime (so only ``lanes`` lane-scoped checkouts/runners are ever needed), pull
+    bundles off a shared queue and run the unattended ``driver.run_issue``.
+    """
+    if cfg.lanes <= 1 or len(bundles) <= 1:
+        for d in bundles:
+            def _build(d=d):
+                _plan_if_unplanned(cfg, d, None)  # iterate-plan may have re-opened it
+                driver.run_issue(d, cfg)
+            _isolate(d, "build/check", _build)
+        return
+
+    # Serial Plan pre-pass — the interactive Plan beat stays out of the pool.
+    for d in bundles:
+        _isolate(d, "plan", lambda d=d: _plan_if_unplanned(cfg, d, None))
+    # Pooled drive — fixed lane slot per worker; gates read it via lane.current().
+    work = deque(bundles)
+    lock = threading.Lock()
+
+    def worker(slot: int) -> None:
+        lane.set_current(slot)
+        while True:
+            with lock:
+                if not work:
+                    return
+                d = work.popleft()
+            _isolate(d, "build/check", lambda d=d: driver.run_issue(d, cfg))
+
+    threads = [threading.Thread(target=worker, args=(k,), name=f"pdca-lane{k}")
+               for k in range(min(cfg.lanes, len(bundles)))]
+    for t in threads:
+        t.start()
+    for t in threads:
+        t.join()
+
+
 # ----------------------------------------------------------------------------
 # Shared multi-bundle driver: build all → cheap-first sign-off → publish → Act once.
 # ----------------------------------------------------------------------------
@@ -193,11 +242,8 @@ def _drive_and_act(
         # Build-all (unattended): advance each bundle to AWAITING_SIGNOFF / COMPLETE.
         # Each bundle is isolated — one that raises (a leaf left it half-written) is
         # skipped this pass, never crashing the sweep and losing the others' progress.
-        for d in bundles:
-            def _build(d=d):
-                _plan_if_unplanned(cfg, d, None)  # iterate-plan may have re-opened it
-                driver.run_issue(d, cfg)
-            _isolate(d, "build/check", _build)
+        # Serial by default; fans out across cfg.lanes lanes when configured (docs 09).
+        _build_all(cfg, bundles)
         # Sign-off, cheap-first, restricted to this batch. ONE interactive session
         # per chunk (≤ SIGNOFF_BATCH_SIZE) walks several bundles — like batch Plan —
         # then every decision is recorded FIRST (apply_now=False) so an iterate-do