Improve concurrency handling, fix part of CI pipeline and more

.devcontainer/README.md

@@ -5,6 +5,15 @@ This directory contains the DevContainer configuration for developing FineCode i
 - `devcontainer.json`: Main configuration file for VS Code DevContainers.
 - `docker-compose.devcontainer.yml`: Compose service definition for the main workspace container.
 
+## Node.js
+
+Node.js 22 is a runtime dependency of `setup_system` handlers that install
+npm-distributed tools (e.g. pi coding agent, which rejects Node older than
+22.19.0). The base image already ships `nvm` itself (no Node version installed), so
+`setup-node.sh` installs Node 22 through that pre-existing `nvm` in
+`postCreateCommand`. Because Node is installed via nvm under
+the `vscode` user, `npm install -g` works without sudo.
+
 ## Local observability stack (opt-in)
 
 The devcontainer includes the repository-level `docker-compose.otel.yml`, but all of
@@ -24,6 +33,26 @@ WAL events are recorded on disk regardless of whether the stack is running, so y
 bring it up later and ingest the history retroactively. See
 [ADR-0052](../../finecode_internal_docs/adr/0052-observability-stack-opt-in-via-compose-profile.md).
 
+## Persistent WM server
+
+`FINECODE_WM_AUTOSTART=1` is set in `.env.example`, so `postStartCommand` runs
+`start-wm-server --detach --keep-alive` on every container start, via
+`start-wm-server.sh`. The workspace stays warm across commands instead of rebuilding
+its config and runners each time. Comment the variable out to go back to a server
+per client; the script is also a no-op when the `dev_workspace` venv does not exist
+yet.
+
+Changing the variable needs the container **recreated**, not reopened — Compose
+resolves `.env` into a container's environment only when that container is created
+(the same caveat as `FINECODE_OTLP_ENDPOINT` above; see [Developing
+FineCode](../docs/guides/developing-finecode.md#local-observability-stack) for the
+rebuild commands).
+
+What comes with a keep-alive server — resident extension runners, a fixed log level,
+and why a server started lazily after a crash is not one — is described under
+[`start-wm-server`](../docs/cli.md#start-wm-server). Re-run
+`sh .devcontainer/start-wm-server.sh` to get the persistent one back.
+
 ## Optional private internal docs mount
 
 The workspace service supports an optional bind mount for private internal docs.

.devcontainer/devcontainer.json

@@ -6,7 +6,28 @@
     ],
     "service": "workspace",
     "shutdownAction": "stopCompose",
-    "postCreateCommand": "sh .devcontainer/post-create-dev-workspace.sh && sh .devcontainer/setup-shell.sh",
+    // Node.js is required by the pi coding agent, which setup_system installs via
+    // npm. Pi's preflight rejects anything older than 22.19.0, so Debian's own
+    // nodejs package is not an option, and so is the node devcontainer feature: the
+    // base image already ships nvm itself, and running the feature on top of that
+    // hits a BuildKit bug writing into nvm's cache dir. setup-node.sh installs Node
+    // via that pre-existing nvm instead, at container-create time. See
+    // setup-node.sh for the full explanation.
+    // remoteEnv (not containerEnv): containerEnv is baked into the compose
+    // service's own environment, including its `sleep infinity` entrypoint process —
+    // and "${containerEnv:PATH}" does not self-resolve there, so it overwrote PATH
+    // with the literal unexpanded string and broke everything, including `sleep`.
+    // remoteEnv only applies to devcontainer-managed processes (postCreateCommand,
+    // terminals), where the substitution does work.
+    "remoteEnv": {
+        "NVM_DIR": "/usr/local/share/nvm",
+        "PATH": "/usr/local/share/nvm/current/bin:${containerEnv:PATH}"
+    },
+    "postCreateCommand": "sh .devcontainer/setup-node.sh && sh .devcontainer/post-create-dev-workspace.sh && sh .devcontainer/setup-shell.sh",
+    // postStart, not postCreate: a stopped container loses its WM server, so this
+    // has to run again on every start. The venv it needs may not exist yet on the
+    // first create; the script exits cleanly in that case.
+    "postStartCommand": "sh .devcontainer/start-wm-server.sh",
     "customizations": {
         "vscode": {
             "extensions": [

.devcontainer/docker-compose.devcontainer.yml

@@ -11,6 +11,9 @@ services:
       # both COMPOSE_PROFILES=otel and FINECODE_OTLP_ENDPOINT in .env — the ready
       # collector URL is provided there, so developers never compose it themselves.
       - FINECODE_OTLP_ENDPOINT=${FINECODE_OTLP_ENDPOINT:-}
+      # Read only by .devcontainer/start-wm-server.sh; passed through here so the
+      # value set in .env is visible to processes inside the container.
+      - FINECODE_WM_AUTOSTART=${FINECODE_WM_AUTOSTART:-}
       # uv's default cache (~/.cache/uv) lives on the container's overlay
       # filesystem, a different device than this bind-mounted workspace. uv
       # dedupes installs by hardlinking from its cache into each venv, but

.devcontainer/setup-node.sh

@@ -0,0 +1,24 @@
+#!/bin/sh
+# Installs Node via the nvm already baked into the base image.
+#
+# We do NOT use the ghcr.io/devcontainers/features/node feature for this:
+# the base image already ships nvm itself (no Node version installed yet)
+# at /usr/local/share/nvm, owned by vscode:nvm. Running the node feature on
+# top of that hits its "NVM already installed" code path, which skips the
+# ownership fixup it only does for a freshly created nvm dir and calls
+# `nvm install` directly. That specific command works fine under a normal
+# `docker run`, but fails with "Permission denied" writing into
+# /usr/local/share/nvm/.cache when run inside a BuildKit RUN layer, which
+# does not reproduce ownership from the base image's read-only layer
+# correctly on copy-up. Running nvm install here, in postCreateCommand
+# (a normal container process, not a build layer) sidesteps the bug.
+set -eu
+
+export NVM_DIR="/usr/local/share/nvm"
+export NVM_SYMLINK_CURRENT=true
+# shellcheck source=/dev/null
+. "$NVM_DIR/nvm.sh"
+
+umask 0002
+nvm install 22
+nvm alias default 22

.devcontainer/start-wm-server.sh

@@ -0,0 +1,29 @@
+#!/bin/sh
+# Start a persistent shared FineCode WM server at container start, so the
+# workspace stays warm for the life of the container. What that costs is
+# documented in docs/cli.md, "Autostarting a persistent server".
+#
+# FineCode itself never reads FINECODE_WM_AUTOSTART — this script is its only
+# consumer, and it turns it into an explicit `--keep-alive` on the server it
+# starts, because keep-alive must never be ambient.
+set -eu
+
+if [ -z "${FINECODE_WM_AUTOSTART:-}" ] || [ "${FINECODE_WM_AUTOSTART}" = "0" ]; then
+    exit 0
+fi
+
+VENV_PYTHON=".venvs/dev_workspace/bin/python"
+if [ ! -x "$VENV_PYTHON" ]; then
+    echo "FineCode WM autostart: $VENV_PYTHON not found, skipping." >&2
+    exit 0
+fi
+
+# Never fatal: the workspace is fully usable without it, because every client
+# starts a server on demand anyway. Failing container start over a warm cache
+# would be a worse outcome than the cold start it is trying to avoid.
+if "$VENV_PYTHON" -m finecode start-wm-server --detach --keep-alive; then
+    echo "FineCode WM autostart: shared server running."
+else
+    echo "FineCode WM autostart: no shared server reachable yet; it may still be" \
+        "starting, and clients attach to it when it is." >&2
+fi

.env.example

@@ -20,3 +20,9 @@ DEVCONTAINER_INTERNAL_DOCS_PATH=
 # use it to start/stop the collector while the endpoint stays armed.
 # COMPOSE_PROFILES=otel
 # FINECODE_OTLP_ENDPOINT=http://otel-lgtm:4317
+
+# Start a persistent shared WM server when the devcontainer starts, so config and
+# extension runners survive between commands. Comment out to get a server per
+# client instead. What comes with it: docs/cli.md, "Autostarting a persistent
+# server".
+FINECODE_WM_AUTOSTART=1

.github/ci/finecode-user.ci.toml

@@ -0,0 +1,12 @@
+# CI counterpart to a developer's gitignored finecode-user.toml.
+#
+# The `.ci` infix is load-bearing: .gitignore ignores the unanchored name
+# `finecode-user.toml` at any depth, so a file here named exactly
+# `finecode-user.toml` could never be committed. This name is not ignored, which
+# is what lets the public repo track the CI copy while the developer copy stays
+# out of git. Do not "tidy" the infix away.
+
+presets = [{ source = "fine_knowledge" }]
+
+[dependency-groups]
+dev_workspace = ["fine_knowledge~=0.1.0a0"]

.github/workflows/ci-cd.yml

@@ -34,7 +34,10 @@ env:
 jobs:
   build:
     runs-on: ${{ matrix.os }}
-    timeout-minutes: 40
+    # A cold `prepare-envs` over the whole workspace plus the full check suite does
+    # not fit in 40 minutes on the slower matrix legs. The venvs cache keeps the
+    # happy path short; this budget only covers the cold run.
+    timeout-minutes: 120
     strategy:
       fail-fast: false
       matrix:
@@ -71,7 +74,7 @@ jobs:
           fi
 
       - name: Set up Python ${{ env.DEV_WORKSPACE_PYTHON_VERSION }}
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@v6
         with:
           python-version: ${{ env.DEV_WORKSPACE_PYTHON_VERSION }}
 
@@ -81,8 +84,9 @@ jobs:
       # skip reinstalling whenever a restored venv already looks valid, so a stale
       # partial-match restore could mask a dependency that was added since the cache
       # was written.
-      - name: Cache all venvs
-        uses: actions/cache@v4
+      - name: Restore venvs cache
+        id: venvs_cache
+        uses: actions/cache/restore@v5
         with:
           path: |
             .venvs
@@ -132,14 +136,13 @@ jobs:
         if: ${{ !cancelled() && steps.install.outcome == 'success' }}
         run: |
           source .venvs/dev_workspace/${{ matrix.venv_bin }}/activate
-          # TODO: test with all supported python versions
           python -m finecode run --log-level="$FINECODE_LOG_LEVEL" run_tests
         shell: bash
 
       - name: Publish to TestPyPI and verify
         if: runner.os == 'Linux' && github.event_name == 'workflow_dispatch' && inputs.publish_testpypi
         env:
-          FINECODE_CONFIG_PUBLISH_AND_VERIFY_ARTIFACT__INIT_REPOSITORY_PROVIDER__REPOSITORIES: '[{"name": "testpypi", "url": "https://test.pypi.org/"}]'
+          FINECODE_CONFIG_PUBLISH_AND_VERIFY_ARTIFACT__INIT_REPOSITORY_PROVIDER__REPOSITORIES: '[{"name": "testpypi", "index_url": "https://test.pypi.org/simple/", "upload_url": "https://test.pypi.org/legacy/"}]'
           FINECODE_CONFIG_PUBLISH_AND_VERIFY_ARTIFACT__INIT_REPOSITORY_PROVIDER__CREDENTIALS_BY_REPOSITORY: '{"testpypi": {"username": "${{ secrets.TESTPYPI_USERNAME }}", "password": "${{ secrets.TESTPYPI_PASSWORD }}"}}'
         run: |
           source .venvs/dev_workspace/${{ matrix.venv_bin }}/activate
@@ -154,7 +157,7 @@ jobs:
       - name: Publish to PyPI and verify
         if: runner.os == 'Linux' && startsWith(github.ref, 'refs/tags/')
         env:
-          FINECODE_CONFIG_PUBLISH_AND_VERIFY_ARTIFACT__INIT_REPOSITORY_PROVIDER__REPOSITORIES: '[{"name": "pypi", "url": "https://pypi.org/"}]'
+          FINECODE_CONFIG_PUBLISH_AND_VERIFY_ARTIFACT__INIT_REPOSITORY_PROVIDER__REPOSITORIES: '[{"name": "pypi", "index_url": "https://pypi.org/simple/", "upload_url": "https://upload.pypi.org/legacy/"}]'
           FINECODE_CONFIG_PUBLISH_AND_VERIFY_ARTIFACT__INIT_REPOSITORY_PROVIDER__CREDENTIALS_BY_REPOSITORY: '{"pypi": {"username": "${{ secrets.PYPI_USERNAME }}", "password": "${{ secrets.PYPI_PASSWORD }}"}}'
         run: |
           # TODO: make sure git tag exists (for manual trigger)
@@ -170,8 +173,170 @@ jobs:
 
       # TODO: try to replace by finecode action
       - name: Store the distribution packages
-        uses: actions/upload-artifact@v5
+        uses: actions/upload-artifact@v6
         if: runner.os == 'Linux'
         with:
           name: python-package-distributions
           path: dist/
+
+      # Save the venvs explicitly instead of relying on actions/cache's own post-step:
+      # that post-step skips the save when the job fails, which throws away a perfectly
+      # good cold install just because a later check or test failed. The only
+      # precondition here is that the install itself succeeded; on an exact key match
+      # there is nothing new to save.
+      # The key is reused from the restore step, never recomputed: by now every .venvs/
+      # is populated, so a `**/pyproject.toml` glob would walk all of site-packages
+      # (hashFiles times out at 120s) and could hash venv-internal files the restore
+      # key never saw, saving under a key no later run would ever restore.
+      - name: Save venvs cache
+        if: ${{ always() && steps.install.outcome == 'success' && steps.venvs_cache.outputs.cache-hit != 'true' }}
+        uses: actions/cache/save@v5
+        with:
+          path: |
+            .venvs
+            **/.venvs
+          key: ${{ steps.venvs_cache.outputs.cache-primary-key }}
+
+  audit-private:
+    name: Audit (private layer)
+    runs-on: ubuntu-24.04
+    # This job pays a cold prepare-envs over ~74 projects and then a
+    # workspace-wide audit_code, which is slow by design (see
+    # docs/guides/developing-finecode.md "Running checks"), so it carries its own
+    # budget rather than inheriting the build matrix's.
+    timeout-minutes: 60
+    # `secrets` is NOT available in jobs.<job_id>.if -- only github, needs,
+    # vars, inputs. An unavailable context evaluates to empty, so a secrets test
+    # here would skip the job forever without ever erroring. Fork PRs are
+    # excluded with the `github` context alone; the credentials test is the
+    # `env` gate below, which step `if`s can read.
+    if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
+    env:
+      # jobs.<job_id>.env DOES have the secrets context, and steps' `if` has
+      # `env`. This one line is what bridges the two.
+      HAS_PRIVATE_CLONE_APP: ${{ secrets.PRIVATE_CLONE_APP_ID != '' && secrets.PRIVATE_CLONE_APP_PRIVATE_KEY != '' }}
+    steps:
+      - uses: actions/checkout@v5
+        with:
+          # setuptools-scm uses tags to get the current version, fetch history and tags
+          # to get correct version
+          fetch-depth: 0
+          fetch-tags: 'true'
+
+      - name: Report skip reason
+        if: env.HAS_PRIVATE_CLONE_APP != 'true'
+        run: echo "::notice::Private-layer audit skipped - App credentials are not available on this run."
+
+      - name: Determine FineCode log level
+        if: env.HAS_PRIVATE_CLONE_APP == 'true'
+        run: |
+          # WM/ER diagnostic logs are streamed to the job log (verbose is auto-enabled
+          # in CI). Keep them at INFO normally; raise to DEBUG only when the job is
+          # re-run with "Enable debug logging" (GitHub sets RUNNER_DEBUG=1). This keeps
+          # the debug-vs-info decision in CI config — FineCode just honors --log-level.
+          if [ "${RUNNER_DEBUG:-0}" = "1" ]; then
+            echo "FINECODE_LOG_LEVEL=DEBUG" >> "$GITHUB_ENV"
+          else
+            echo "FINECODE_LOG_LEVEL=INFO" >> "$GITHUB_ENV"
+          fi
+
+      - name: Set up Python ${{ env.DEV_WORKSPACE_PYTHON_VERSION }}
+        if: env.HAS_PRIVATE_CLONE_APP == 'true'
+        uses: actions/setup-python@v6
+        with:
+          python-version: ${{ env.DEV_WORKSPACE_PYTHON_VERSION }}
+
+      - name: Mint private-clone token
+        id: app_token
+        if: env.HAS_PRIVATE_CLONE_APP == 'true'
+        uses: actions/create-github-app-token@v3
+        with:
+          app-id: ${{ secrets.PRIVATE_CLONE_APP_ID }}
+          private-key: ${{ secrets.PRIVATE_CLONE_APP_PRIVATE_KEY }}
+          owner: finecode-dev
+          repositories: fine_knowledge,finecode_internal_experiments
+
+      - name: Check out fine_knowledge
+        if: env.HAS_PRIVATE_CLONE_APP == 'true'
+        uses: actions/checkout@v5
+        with:
+          repository: finecode-dev/fine_knowledge
+          path: presets/fine_knowledge
+          token: ${{ steps.app_token.outputs.token }}
+          # A `git clone` with the token in the URL writes it into the clone's
+          # .git/config; checkout with persist-credentials disabled does not.
+          persist-credentials: false
+
+      - name: Check out internal experiments
+        if: env.HAS_PRIVATE_CLONE_APP == 'true'
+        uses: actions/checkout@v5
+        with:
+          repository: finecode-dev/finecode_internal_experiments
+          path: finecode_internal_experiments
+          token: ${{ steps.app_token.outputs.token }}
+          persist-credentials: false
+
+      - name: Install the CI private-layer config
+        if: env.HAS_PRIVATE_CLONE_APP == 'true'
+        run: cp .github/ci/finecode-user.ci.toml finecode-user.toml
+
+      - name: Restore venvs cache
+        id: venvs_cache
+        if: env.HAS_PRIVATE_CLONE_APP == 'true'
+        uses: actions/cache/restore@v5
+        with:
+          path: |
+            .venvs
+            **/.venvs
+          key: ${{ runner.os }}-private-venvs-${{ hashFiles('**/pyproject.toml', '**/preset.toml', 'finecode-user.toml') }}
+
+      - name: Install dependencies
+        id: install
+        if: env.HAS_PRIVATE_CLONE_APP == 'true'
+        run: |
+          # CI must exercise this branch's local source, so finecode and its sibling
+          # packages need an editable install, not a released version from PyPI — see
+          # docs/guides/developing-finecode.md#continuous-integration.
+          sh scripts/setup-dev-workspace.sh
+          source .venvs/dev_workspace/bin/activate
+        shell: bash
+
+      - name: Inspect code
+        if: env.HAS_PRIVATE_CLONE_APP == 'true'
+        run: |
+          source .venvs/dev_workspace/bin/activate
+          python -m finecode run --log-level="$FINECODE_LOG_LEVEL" inspect_code
+        shell: bash
+
+      - name: Extract knowledge
+        if: env.HAS_PRIVATE_CLONE_APP == 'true'
+        run: |
+          source .venvs/dev_workspace/bin/activate
+          python -m finecode run --log-level="$FINECODE_LOG_LEVEL" extract_knowledge
+        shell: bash
+
+      - name: Audit code
+        if: env.HAS_PRIVATE_CLONE_APP == 'true'
+        run: |
+          source .venvs/dev_workspace/bin/activate
+          python -m finecode run --log-level="$FINECODE_LOG_LEVEL" audit_code
+        shell: bash
+
+      - name: Run unit tests
+        if: env.HAS_PRIVATE_CLONE_APP == 'true'
+        run: |
+          source .venvs/dev_workspace/bin/activate
+          python -m finecode run --log-level="$FINECODE_LOG_LEVEL" run_tests
+        shell: bash
+
+      # See the matching step in the `build` job for the rationale: save the venvs
+      # whenever the install succeeded, even if a later check, audit or test failed,
+      # and reuse the restore step's key rather than recomputing it.
+      - name: Save venvs cache
+        if: ${{ always() && env.HAS_PRIVATE_CLONE_APP == 'true' && steps.install.outcome == 'success' && steps.venvs_cache.outputs.cache-hit != 'true' }}
+        uses: actions/cache/save@v5
+        with:
+          path: |
+            .venvs
+            **/.venvs
+          key: ${{ steps.venvs_cache.outputs.cache-primary-key }}