feat(DLD-1456): add manager-visible runtime recovery endpoints - #2
Open
nydamon wants to merge 826 commits into
Open
feat(DLD-1456): add manager-visible runtime recovery endpoints#2nydamon wants to merge 826 commits into
nydamon wants to merge 826 commits into
Conversation
Align with e2e.yml and ensure CI tests exactly the committed dependency tree. The pr-policy job already blocks lockfile changes in PRs, so frozen-lockfile is safe here. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
WIP: routines management, triggers, and execution flow
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
…d and adapterLabels Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…d-home-followups Improve worktree merge/import followups
…d-home-followups fix: restore post-merge route verification
…ge-history-migrations fix: renumber worktree merge history migrations
ci: run e2e tests on PRs
…resh-automerge-guard fix(ci): skip auto-merge step when lockfile is unchanged
Merge pr-verify.yml, pr-policy.yml, and pr-e2e.yml into a single pr.yml with three parallel jobs (policy, verify, e2e). Benefits: - Single concurrency group cancels all jobs on new push - Consistent Node 24 across all jobs - One file to maintain instead of three The jobs still run independently (no artifact sharing) since pnpm cache makes install fast and the upload/download overhead for node_modules would negate the savings. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…r-workflows ci: consolidate PR workflows into a single file
…-bootstrap feat(evals): bootstrap promptfoo eval framework (Phase 0)
…issues-by-project feat(ui): add project filter to issues list
…image-attachments fix: embed uploaded images inline in comments via paperclip button
Handle GitHub pull_request webhook events to auto-update linked Paperclip issue status on lifecycle transitions: - opened / reopened → in_progress - closed + merged → done (with merged-by comment) - closed, not merged → blocked Changes: - constants.ts: add "pull_request" to SUPPORTED_GITHUB_EVENTS - github-types.ts: add GitHubPullRequestEvent type and include in union - worker.ts: add handlePullRequestEvent() and register pull_request case Co-Authored-By: Paperclip <noreply@paperclip.ing>
The plugin SDK issues.update() type does not include a comment field. Post the PR event comment separately via ctx.issues.createComment(). Co-Authored-By: Paperclip <noreply@paperclip.ing>
feat: upgrade to upstream v2026.325.0 + revert npm scope to @paperclipai
…erclipai#86) * fix(github-plugin): harden delivery dedup, search pagination, and comment fetching - Replace unbounded per-delivery state keys with a bounded ring buffer (200 entries, 24h TTL) stored in a single state key. The plugin SDK has no list/scan or TTL support, so individual keys would accumulate forever. - Add configurable maxResults parameter (1-100) to github_search_issues tool, defaulting to 10. - Add automatic pagination to listComments (up to 1000 comments across 10 pages) so issues with many comments don't silently lose data. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(authz): allow CEO agent to cancel heartbeat runs Widen POST /heartbeat-runs/:runId/cancel to accept CEO-role agents in addition to board users. Adds assertBoardOrCeoAgent helper that does a lightweight role check, company-scoped access guard, and proper actor attribution in activity logs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
paperclipai#87) Commit 01c56cd added `"private": true` to all workspace packages during the @paperclipai_dld → @paperclipai scope rename. This caused the Release workflow to fail with "no public packages were found in the workspace." because release-package-map.mjs only considers non-private packages. Removes the flag from all 12 packages that had it accidentally added, restoring the pre-merge publish behavior. Fixes: DLD-1036, DLD-1035 Co-authored-by: Senior Platform Engineer <noreply@paperclip.ing>
…paperclipai#88) The npm registry occasionally returns a 404 for PUT requests on first publish to a new scope or due to registry propagation lag, even when the package is successfully queued/published. With set -euo pipefail this caused the publish loop to abort early, leaving later packages (including @paperclipai/cli) unpublished and the git tag uncreated. After a publish failure, we now verify the version is actually available on npm (6 attempts, 5s delay). If confirmed, we warn and continue. If not found after retries, we fail with a clear error. Fixes Release paperclipai#31 CI failure (DLD-1041). Co-authored-by: Senior Platform Engineer <noreply@paperclip.ing>
…params (paperclipai#89) Switch Step 5 from `pnpm publish` to `npm publish` to avoid pnpm wrapper issues with scoped package auth. pnpm 9.15.4 passes `--no-git-checks` to npm which doesn't recognise it, and may handle auth differently for PUT requests on scoped packages. Also use NPM_PUBLISH_VERIFY_ATTEMPTS and NPM_PUBLISH_VERIFY_DELAY_SECONDS (already set in release.yml for CI) as the window for post-failure verification, matching the patience used in Step 6. Co-authored-by: Senior Platform Engineer <noreply@paperclip.ing>
…ai#90) PR paperclipai#87 removed "private": true from 12 packages, causing the release workflow to attempt publishing to @paperclipai on npm — a scope we don't own. PRs paperclipai#88 and paperclipai#89 tried to work around the resulting 404 but the root cause is that we don't publish to npm at all. Restores the invariant documented in CLAUDE.md: all packages use upstream's @paperclipai scope and are marked private. Reverts the package.json changes from paperclipai#87. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…inventory - Add pi-autoresearch team experiment framework (docs/pi-autoresearch-framework.md) - Document plugin survival across deploys (built-in, volume-installed, CLI extensions) - Document superpowers skills export procedure - Document pi-autoresearch extension with reinstall instructions - Update secrets inventory: add Viracue Stripe keys (test + live), separate working vs undecryptable secrets, remove stale entries Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…inventory (paperclipai#91) - Add pi-autoresearch team experiment framework (docs/pi-autoresearch-framework.md) - Document plugin survival across deploys (built-in, volume-installed, CLI extensions) - Document superpowers skills export procedure - Document pi-autoresearch extension with reinstall instructions - Update secrets inventory: add Viracue Stripe keys (test + live), separate working vs undecryptable secrets, remove stale entries Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…pattern
The `logChain = logChain.then(…)` pattern in `runChildProcess()` created an
ever-growing linked list of promise closures, each retaining the log chunk
text. For agents producing large output (Research Agent: 257MB in 30min),
this kept all chunks in memory until process exit, causing the server to OOM
at ~4.6GB heap after processing ~35 runs.
Changes:
- Replace promise chain with a shift-based queue that allows GC of processed
entries immediately after `onLog()` resolves
- Cap `executeProcess()` stdout/stderr buffers at 4MB (previously unbounded)
- Use `.once("exit")` instead of `.on("exit")` in `registerRuntimeService()`
to prevent listener closure retention after child exit
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…mory-leak fix(memory): OOM crash from unbounded log promise chain
15-18 agents heartbeating simultaneously spawn CLI processes (claude, opencode, pi) that collectively exceed the 6GB container memory limit. Each CLI process uses 230-350MB, and the server itself grows to 2.3GB+ under load. Add HEARTBEAT_GLOBAL_MAX_CONCURRENT_RUNS (default: 3, configurable via env var). When the limit is reached, excess runs stay queued and are picked up on the next heartbeat tick (~30s) or when a slot frees up via the per-agent queue promotion that already runs after each run completes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…-concurrency-limit fix(heartbeat): global concurrent run limit to prevent OOM
…pai#94) The adapter-utils package was never explicitly built in either Dockerfile, so changes to its TypeScript source (like the PR paperclipai#92 log chain memory fix) were not compiled into the dist/ used at runtime. The pre-existing dist/ from pnpm install was used instead, containing the old leaking code. Add `pnpm --filter @paperclipai/adapter-utils build` to both Dockerfiles before the server build step. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…anup (paperclipai#95) - Lower --max-old-space-size from 4608MB to 2048MB so V8 GCs aggressively and leaves room for child CLI processes (1-1.5GB) within the 6GB container limit. - Reduce plugin session event subscription timeout from 30min to 5min to prevent listener accumulation on the live-events EventEmitter. - Set maxListeners to 50 (was unlimited) so Node.js warns before subscriptions spiral. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…pai#96) When all packages have \"private\": true (as restored in PR paperclipai#90), the nightly canary release was failing with exit code 1 instead of skipping cleanly. This caused recurring DLD CI-failure issues (DLD-1201, DLD-1202) on every scheduled run. Change the hard fail to a graceful info+exit 0 so the CI job succeeds as a no-op when this fork has no publishable packages. Co-authored-by: Senior Platform Engineer <platform@viraforge.ai> Co-authored-by: Paperclip <noreply@paperclip.ing>
The 2GB limit from PR paperclipai#95 caused 7 restarts: V8 GC stats show the server legitimately needs ~2GB heap for context resolution, DB queries, and adapter orchestration across 17 concurrent agents. 2GB left no headroom for allocation spikes. 3GB balances server needs (2-2.5GB typical, 3GB peak) with child process memory (3 concurrent × ~600MB = 1.8GB) within the 6GB container limit. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Exposes process.memoryUsage() and V8 heap space statistics for live memory monitoring. Heapdump endpoint writes a V8 heap snapshot to disk for offline analysis of what objects are consuming the 3GB+ heap. These are essential for diagnosing the remaining memory leak that persists after the logChain fix (PR paperclipai#92), concurrency limit (PR paperclipai#93), and subscription timeout tightening (PR paperclipai#95). Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
) * fix(oom): add /api/health/memory and /api/health/heapdump debug endpoints Exposes process.memoryUsage() and V8 heap space statistics for live memory monitoring. Heapdump endpoint writes a V8 heap snapshot to disk for offline analysis of what objects are consuming the 3GB+ heap. These are essential for diagnosing the remaining memory leak that persists after the logChain fix (PR paperclipai#92), concurrency limit (PR paperclipai#93), and subscription timeout tightening (PR paperclipai#95). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(heartbeat): cap resultJson before DB storage to prevent OOM Adapters store full stdout/stderr (up to 4 MB each) in resultJson, which gets written to the heartbeat_runs table. These multi-MB strings land in V8's large_object_space (objects >256 KB) and are the primary driver of the OOM crash loop — production DB shows rows with 4.7 MB resultJson while large_object_space consumes 669 MB (71% of heap). Changes: - Add capResultJsonForStorage() that truncates any string field >8 KB to its last 8 KB tail. Full output is already persisted in NDJSON run-log files and stdoutExcerpt/stderrExcerpt columns. - Apply capResultJsonForStorage at the DB write point in executeRun. - Null out adapterResult.resultJson after the DB write to allow V8 GC before the long executeRun scope ends. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…a, watchdog (DLD-1009)
B1 — Schema migration (0045):
- Add last_heartbeat_at TIMESTAMPTZ DEFAULT NOW() to heartbeat_runs
- Add index on (status, last_heartbeat_at) for efficient watchdog queries
B2 — Ping endpoint:
- POST /api/agents/:agentId/runs/:runId/ping
- Agent JWT auth; caller must own the run
- Updates last_heartbeat_at and returns { lastHeartbeatAt: ISO-8601 }
- 404 if not found, 403 if not owner, 409 if not running
B4 — Watchdog cron:
- heartbeatService.timeoutStaleRuns() — marks running runs timed_out
if last_heartbeat_at < NOW() - PAPERCLIP_RUN_TIMEOUT_MINUTES (default 30)
- Skips runs with active in-process handles (runningProcesses / activeRunExecutions)
- Releases locked issue execution and promotes next queued run
- Logs to activity_log with actorType=system, action=heartbeat.timed_out
- Wired into existing heartbeatSchedulerIntervalMs setInterval in index.ts
Co-Authored-By: Paperclip <noreply@paperclip.ing>
…gent PASS (DLD-1281) Adds @paperclipai/plugin-qa-gate: a Paperclip plugin that listens for issue.updated events and blocks agents from setting status to done unless a @qa-agent PASS comment exists in the thread. Bypass rules: - Board users (actorType=user) always bypass the gate - Issues labelled no-code, research, docs, or backlog are exempt 10 unit tests pass (createTestHarness). Plugin added to bundled examples list in server/src/routes/plugins.ts for one-click board installation. Co-Authored-By: Paperclip <noreply@paperclip.ing>
The POST /api/plugins/:pluginId/enable route was calling lifecycle.enable() for all statuses, but lifecycle.enable() only accepts disabled/error/ upgrade_pending. Plugins stuck in 'installed' status needed lifecycle.load() to transition to ready. Also widens AvailablePluginExample.tag to include "automation" for the QA Gate bundled plugin entry, and fixes a doc comment that incorrectly said disable transitions to 'installed' (it's 'disabled'). Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Update POST /plugins/:pluginId/enable and /disable to use assertBoardOrCeoAgent instead of assertBoard, enabling the CEO agent to call these endpoints without board access. assertBoardOrCeoAgent already exists in authz.ts and is used in agents.ts. Added it to the plugins.ts import. Co-Authored-By: Paperclip <noreply@paperclip.ing>
Expand QA gate bypass signals for operational/non-code tickets to avoid auto-reopening stale cleanup issues into in_review. Add unit regressions and Playwright API E2E verification for both bypass and enforcement paths. Co-Authored-By: Paperclip <noreply@paperclip.ing>
Address QA FAIL feedback by adding mandatory plugin version bump (0.1.1), widening stale-operational bypass matching, and updating the Playwright QA-gate flow suite to deterministic harness-based outcome checks. Co-Authored-By: Paperclip <noreply@paperclip.ing>
… fixes - Add normalizeAgentUrlKey matching in findMentionedAgents so multi-word agent names are reachable via @url-key (25/25 agents vs 7/25 before) - Bump server version 0.3.2 -> 0.3.3 - Fix pre-existing TS errors in heartbeat.ts: - setUTCHours extra arg, errorCode out of scope, null assigneeAgentId - Add in_review assignee validation Co-Authored-By: Paperclip <noreply@paperclip.ing>
…oken agents Co-Authored-By: Paperclip <noreply@paperclip.ing>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a94ecc86e9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+117
to
+120
| if (agent.status !== "paused" && agent.status !== "idle") { | ||
| return res.status(422).json({ | ||
| error: `Cannot trigger heartbeat retry for agent in '${agent.status}' status. Pause the agent first.`, | ||
| }); |
There was a problem hiding this comment.
This handler explicitly allows paused agents and even tells managers to pause first, but the next heartbeat.wakeup(...) call rejects paused agents with a conflict (enqueueWakeup blocks agent.status === "paused"). In practice, /agents/:id/recover/retry-heartbeat will fail for the main recovery case (paused direct reports) and return 409 instead of scheduling a retry.
Useful? React with 👍 / 👎.
| res.json({ status: "retry_scheduled", run }); | ||
| }); | ||
|
|
||
| router.post("/agents/:id/recover/reset-session", validate(managerRecoveryActionSchema), async (req, res) => { |
There was a problem hiding this comment.
Using managerRecoveryActionSchema here drops unknown fields, and that schema only defines reason, so taskKey is stripped before it is read. The endpoint then always calls resetRuntimeSession with taskKey: null, which performs a full session reset instead of targeted task-session cleanup when managers provide a specific task key.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
POST /agents/:id/recover/pause— pause agent + cancel active runsPOST /agents/:id/recover/resume— resume from paused/idlePOST /agents/:id/recover/retry-heartbeat— trigger heartbeat via wakeup()POST /agents/:id/recover/reset-session— clear runtime session statePOST /agents/:id/recover/rollback-config— rollback to config revisionVerification
Related