fix(routines): origin-only fallback for stagnation monitor deduplication (DLD-4090) - #10
Open
nydamon wants to merge 916 commits into
Open
fix(routines): origin-only fallback for stagnation monitor deduplication (DLD-4090)#10nydamon wants to merge 916 commits into
nydamon wants to merge 916 commits into
Conversation
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Infer portable repo metadata from local git workspaces when repoUrl is missing, and collapse repeated task workspace export warnings into a single summary per missing workspace. Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Normalize GitHub skill directories for blob/file imports and when reading legacy stored metadata so imported SKILL.md files resolve correctly. Co-Authored-By: Paperclip <noreply@paperclip.ing>
Documents the `paperclipai company export` and `paperclipai company import` CLI commands, covering package format, all options, target modes, collision strategies, GitHub sources, interactive selection, and API endpoints. Co-Authored-By: Paperclip <noreply@paperclip.ing> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This reverts commit 17876ec1dc65a9150488874d79fc2fcc087c13ae.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Prevent company imports from re-enabling scheduler heartbeats on imported agents and cover both new-company and existing-company import flows in portability tests. Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
…rvice (paperclipai#63) * fix(server): comprehensive postgres.js Date parameter binding throughout heartbeat service postgres.js driver requires ISO string timestamps for parameter binding, not Date objects. This commit comprehensively fixes all 35+ locations in heartbeat.ts where Date objects were being passed to database operations (insert/update) on timestamp fields. Key fixes: - setRunStatus(): Convert updatedAt to ISO string - setWakeupStatus(): Convert updatedAt to ISO string - enqueueProcessLostRetries(): Convert all updatedAt assignments to ISO strings - startNextQueuedRunForAgent(): Convert startedAt and updatedAt to ISO strings - updateRuntimeState(): Convert updatedAt to ISO string - All remaining finishedAt, startedAt, updatedAt assignments throughout service Pattern: finishedAt/startedAt/updatedAt are now converted to new Date().toISOString() as any before passing to .set() and .values() operations. This resolves the 100% CPU issue from periodic heartbeat operations failing with: - ERR_INVALID_ARG_TYPE when Date objects are passed to postgres.js - Type errors when trying to call toISOString() on already-stringified values Fixes: agents stuck in 'queued' status after deployment; periodic heartbeat recovery failures Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix(server): address critical Date parameter binding omissions in heartbeat service Code review identified 3 critical missing timestamp conversions: 1. lastHeartbeatAt in updateRuntimeState() - was passing raw Date object 2. executionLockedAt at lines 3077 and 3266 - were passing raw Date objects 3. startedAt in startNextQueuedRunForAgent() - could pass string inconsistently All timestamp fields now properly converted to ISO strings before postgres.js parameter binding. Added explanatory comments for the postgres.js requirement. Fixes code review findings from PR paperclipai#63 code-reviewer assessment. --------- Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
… recovery (paperclipai#65) Fixed two critical bugs in heartbeat recovery that were preventing server startup: 1. Line 1582: Convert processLostRetryAt to ISO string before database update 2. Line 1605: Use ISO string (nowStr) instead of Date object in SQL where clause These issues were causing 'value.toISOString is not a function' errors during startup heartbeat recovery when reaping orphaned runs and scheduling retries. This is a follow-up to PR paperclipai#63 which didn't catch these timestamp parameter bindings. Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
…ecution workspaces Fixed three remaining Date parameter binding issues in heartbeat service: - Line 1854: Cost event occurredAt field was passing Date object directly - Line 2062: Execution workspace lastUsedAt field in update() call - Line 2093: Execution workspace openedAt and lastUsedAt fields in create() call All timestamp fields now convert to ISO strings before database operations, matching postgres.js requirement for timestamp parameters to be ISO strings rather than Date objects. This resolves "value.toISOString is not a function" errors in the heartbeat recovery system that were causing agents to remain stuck in queued status. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…p-recovery fix: resolve remaining postgres.js Date parameter binding in heartbeat service
PAPERCLIP_MIGRATION_PROMPT=never was checked before PAPERCLIP_MIGRATION_AUTO_APPLY=true, causing auto-apply to never trigger when both env vars are set (as in dev:watch). Swap the check order so AUTO_APPLY takes precedence. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…es service Fixed Date parameter binding in the update() method that was causing "value.toISOString is not a function" errors in heartbeat service. The updatedAt field was passing a raw Date object instead of an ISO string to Drizzle ORM. This was identified as a critical issue during code review of PR paperclipai#66. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…p-recovery fix: resolve postgres.js Date parameter binding in execution-workspaces service
…ness (paperclipai#168) Three CEO behavior fixes: 1. Handoff cooldown: CEO must not post follow-up comments on issues within 15 minutes of reassigning them. The generic timer heartbeat was posting "please prioritize" nudges immediately after handoff, wasting tokens while the new assignee was still picking up the work. 2. Forbidden operations table: Explicit list of operations the CEO must never attempt (SSH, GitHub, git push, Docker, code writing) with delegation instructions. Prevents the 10+ failed SSH attempts pattern observed on DLD-1783. 3. Gstack browse availability: Documents that the headless browser IS available in the container runtime, both for CEO (strategic checks only) and all agents (full QA testing). Prevents false "browser not available" claims. Also adds handoff discipline section to CEO AGENTS.md and headless browser documentation to the default AGENTS.md for all agents. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…perclipai#167) * fix(heartbeat): cast issueId text to uuid in sweeper join The contextSnapshot->>'issueId' returns text but issues.id is uuid, causing "operator does not exist: text = uuid" on production. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: auto-deploy on push, adapter circuit breaker, plugin severity tiers - deploy-vultr.yml: auto-trigger on push to master (no longer manual-only) - docker.yml: tags-only trigger (general image was redundant with deploy-vultr) - heartbeat: adapter_failed circuit breaker pauses agent after 2 consecutive adapter failures instead of retrying indefinitely and burning tokens - heartbeat: process_lost exhaustion now logs activity on the issue feed for visibility instead of only appearing in run events - github plugin: workflow severity tiers (critical/standard/informational) with configurable per-workflow mapping - github plugin: root-cause escalation creates diagnostic issue after 3 failures of the same workflow within 24h sliding window - CLAUDE.md: updated deploy documentation to reflect auto-deploy changes Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(github-plugin): remove dead informational→medium severity mapping AI review correctly identified that the `informational` case in `severityToPriority` was unreachable — informational workflows return early before issue creation. Narrow the type to exclude it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: document adapter_failed circuit breaker in CLAUDE.md AI review noted the circuit breaker behavior should be documented for operator awareness. Added section after the unpicked assignment watchdog docs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(test): clean up activity_log in process recovery test teardown The process_lost consolidation added logActivity() calls that create activity_log rows referencing heartbeat_runs. The test teardown must delete activity_log before heartbeat_runs to avoid FK violations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…ocal adapter (paperclipai#169) CEO agent was burning excessive tokens through wrong API URL guessing (7 retries), missing permission injection, and unbounded 55-turn runs. This brings pi_local to parity with claude_local and adds soft turn limits via timeout derivation. - Inject PAPERCLIP_CAN_ASSIGN_TASKS env var from context (claude_local parity) - Inject paperclipPermissionNote into user prompt (claude_local parity) - Inject API URL note into prompt so LLM doesn't guess external URLs - Add maxTurnsPerRun→effectiveTimeoutSec mapping (2 min/turn) since pi CLI has no --max-turns flag - Use effectiveTimeoutSec in runChildProcess call and timeout error message Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…i#170) * feat(ai-review): add auto-remediation for review findings When the AI review finds non-critical issues (PASS_WITH_NOTES or FAIL), the workflow now automatically attempts to fix them by sending findings and file contents to an LLM, applying patches, validating syntax, and re-reviewing. Critical findings skip remediation and post FAIL directly. Infinite loop prevention: github.token pushes don't trigger PR events, and [ai-fix] commit markers are detected to skip remediation on re-runs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(docker): add unzip to container, document available agent tools Agents waste turns probing for python3, unzip, zip, wget etc. on every run. Add unzip to both Dockerfiles and document available/unavailable tools in CLAUDE.md so agents stop discovery-probing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…erclipai#172) Prevents UUID confabulation by giving agents an accurate roster of all non-terminated company agents (name, ID, role, status) at every heartbeat. Weaker models like MiniMax M2.7 were hallucinating agent UUIDs from stale session memory, causing 404s and failed assignments. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…paperclipai#173) Updates AGENTS.md (root + onboarding) and CLAUDE.md with verification standards requiring engineers to test interactively before handoff and QA to perform outcome-based testing (not just static checks/grepping). Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…perclipai#175) - HIGH_RISK verdict now auto-merges like PASS_WITH_NOTES instead of blocking on human approval (review comment still flags concerns) - When merge fails because branch is behind, automatically calls updateBranch to sync with base — CI re-runs and merge retries - When branch update fails (conflicts), posts a comment explaining manual rebase is needed Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…aperclipai#174) The previous default of 5/hour was too aggressive and blocked legitimate batch issue creation via agent API keys. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
… issues (paperclipai#176) Builds on paperclipai#173's general verification standards by adding specific gstack browse command sequences that engineers and QA agents must follow for any simulation, extension, or live-call issue. Includes required evidence format. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…gured (paperclipai#177) The feedback export system was always creating a share client defaulting to telemetry.paperclip.ing even when no PAPERCLIP_FEEDBACK_EXPORT_BACKEND_URL was set. This caused the 5-second flush timer to run 12 regex redaction patterns on every trace, consuming 56% of CPU at idle (confirmed via V8 CPU profiling). Two traces had accumulated 5600+ failed retry attempts. Now the share client and flush timer are only created when an export backend URL or token is explicitly configured. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…asks (paperclipai#178) * fix(perf): disable feedback telemetry when no export backend is configured The feedback export system was always creating a share client defaulting to telemetry.paperclip.ing even when no PAPERCLIP_FEEDBACK_EXPORT_BACKEND_URL was set. This caused the 5-second flush timer to run 12 regex redaction patterns on every trace, consuming 56% of CPU at idle (confirmed via V8 CPU profiling). Two traces had accumulated 5600+ failed retry attempts. Now the share client and flush timer are only created when an export backend URL or token is explicitly configured. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(heartbeat): enforce task-bound scoping on wake, skip active-run tasks When PAPERCLIP_TASK_ID is set (task-bound wake), agents now MUST work only on that task and exit — no inbox fetch, no reviewing other issues. This prevents duplicate work when multiple agents are woken in parallel. During global heartbeats (no PAPERCLIP_TASK_ID), agents now skip tasks where activeRun is non-null, meaning another run is already executing. This avoids checking out and duplicating work on issues that a wake run is already handling. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…andoff (paperclipai#180) Two fixes: 1. Same-assignee retrigger: when a comment is posted on an issue where the assignee didn't change, fire a task-specific wakeup so the agent wakes with paperclipTaskId set. Previously, CEO "re-triggering" an agent via comment produced no wakeup because assigneeWillChange was false — the agent only picked up the task on the next global heartbeat (without task context), causing it to work on other things instead. 2. Engineer→devops handoff: engineers can now assign issues to devops (Senior Platform Engineer) for infrastructure tasks requiring VPS/SSH/CI access. Previously only engineer→qa was allowed, creating dead ends when engineers needed SPE to perform VPS operations. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…and cross-issue comments (paperclipai#182) Three server-side fixes to reduce heartbeat waste: 1. Consecutive No-Op Circuit Breaker: Track timer runs producing <200 output tokens via `consecutiveNoopCount` on agents. After 3+ consecutive no-ops, tickTimers applies exponential backoff (2x→4x→8x cap). Any productive timer run resets the counter. Non-timer wakes unaffected. 2. Gate-Block Backoff: Track gate blocks via `gateBlockCount` on issues. Incremented at all 6 gate block sites. After 3+ blocks, enqueueWakeup skips issue-specific dispatch. Counter resets on status or assignee change. 3. Cross-Issue Comment Observability: Fire-and-forget detection in POST /issues/:id/comments logs when an agent run comments on an issue it was not dispatched for. Observe-only — no blocking. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…aperclipai#181) The ai-review workflow posts the ai-review/verdict commit status using github.token. GitHub does not emit status events for statuses created by the built-in GITHUB_TOKEN, so the merge-automation workflow (triggered by on: status) never fires. This broke auto-merge for all PRs. Switch the "Post verdict status" step to use REPO_ADMIN_PAT so the status event propagates and merge-automation can pick it up. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…recovery (paperclipai#179) * ops: tune server for 16GB host, add autovacuum tuning, document telemetry + hermes - Bump container mem_limit from 6g to 12g and Node heap from 3072MB to 8192MB to match the actual 16GB host (was sized for an 8GB host that no longer exists) - Add PostgreSQL autovacuum tuning: vacuum at 2% dead tuples (was 20% default), analyze at 1%, check every 30s — prevents the 12-24hr vacuum gaps that caused DB CPU spikes - Document feedback telemetry disable decision (PR paperclipai#177) with root cause and re-enable instructions - Document hermes sidecar architecture and correct management path (docker exec via bridge script, not SSH) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(heartbeat): auto-recover agents from process_lost error state Deploys restart the server container, killing all active agent runs with process_lost. This puts agents into error state where they stay until manually reset — a recurring burden since auto-deploy runs on every push to master. New recoverProcessLostAgents() sweeper runs every ~30s: - Finds agents in error state (no pauseReason) - Checks if last run failed with process_lost - Waits 60s after failure (avoids racing with retry mechanism) - Resets agent to idle automatically Does NOT recover agents paused by the adapter_failed circuit breaker. Also updated Monitor agent instructions to skip process_lost failures entirely — no issues are created for this known platform behavior. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…ation, multi-word mentions (paperclipai#183) - Expand ALLOWED_HANDOFFS to allow all roles to escalate to CEO/CTO - Add devops→[qa,engineer] bidirectional handoff - Add pm, cmo, researcher, general role handoff targets - Add multi-word greedy mention matching (Pass 3) for "@qa Agent" style mentions - 4 new mention tests, 5 new assignment policy tests (40 total passing) - Update AGENTS.md and CLAUDE.md docs Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…i#184) The merge-automation workflow only triggered on `on: status` events (ai-review/verdict). If ai-review posted its verdict before verify finished, merge-automation would defer and never retry — verify completion doesn't emit a status event. Add `check_run: types: [completed]` trigger so the workflow also fires when verify, policy, or review check runs complete. The job now resolves the SHA from either event type and checks all conditions uniformly. Also adds concurrency guard to prevent duplicate merge attempts when multiple events fire for the same commit. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(server): enforce task-bound scope at API level for agent runs Agents woken for a specific issue via assignment are now structurally prevented from reading or writing other issues. This replaces the prompt-level "work ONLY on this task" instruction which models (especially MiniMax M2.7) ignored — causing waste like the QA agent run that triaged 3 unrelated issues consuming 969K input tokens. New utility: task-bound-scope.ts with request-level cached resolution. Guards applied to 22 endpoints (403 for reads, 422 for writes). Exempt: issue creation (sub-tasks) and release (checkout cleanup). Inbox-lite short-circuits to bound issue only. 24 new tests, 12 existing test files updated for getRun mock. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: cast Request through unknown for Symbol index access Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
paperclipai#186) All agents now have SSH access to a dedicated testing VPS (207.148.14.165) with Chromium and Playwright pre-installed. Env vars BROWSER_TEST_HOST, BROWSER_TEST_USER, and BROWSER_TEST_SSH_KEY are wired to all 24 agents. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…lipai#187) Adapted from vercel-labs/agent-browser dogfood skill for Paperclip's Browser Testing VPS infrastructure. Includes issue taxonomy reference and structured report template. Bundled in skills/ → auto-registered as paperclipai/paperclip/dogfood with required=true for all agents. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…aperclipai#194) * fix(cors): restrict Access-Control-Allow-Origin to allowedOrigins whitelist DLD-2013: Replace wildcard CORS header with env-var-driven origin whitelist. - Read ALLOWED_ORIGINS env var (comma-separated origins) - Fall back to empty list (no CORS headers emitted — same-origin only) - Add OPTIONS preflight handler for CORS preflight requests - Plugin UI dev loading (localhost) supported via ALLOWED_ORIGINS env var * fix(cors): guard against undefined origin in allowedOrigins check TypeScript strict mode rejects passing string|undefined to includes(string). Guard with explicit origin && check before the whitelist test. Co-Authored-By: Paperclip <noreply@paperclip.ing> * [SEC-FIX] DLD-2014: Block auto-merge on HIGH_RISK verdict in merge-automation.yml Before: HIGH_RISK verdict was logged but script continued to auto-merge, creating a security bypass where elevated-risk PRs could be auto-merged. After: HIGH_RISK (and HUMAN_REQUIRED) verdicts now return early, blocking auto-merge and requiring human review before any merge can proceed. Acceptance criteria satisfied: - No auto-merge can occur without passing all verdict checks - HIGH_RISK verdicts block merge (DLD-2014) - FAIL verdicts block merge - PASS verdicts proceed only if all CI checks pass --------- Co-authored-by: Senior Platform Engineer <spe@viraforge.local> Co-authored-by: Paperclip <noreply@paperclip.ing>
…lipai#191) Agents ping-ponging via comments on the same issue (e.g. DLD-2003 with 5 consecutive CTO/CEO touches) could loop indefinitely because existing defenses (consecutiveNoopCount, gateBlockCount) don't catch wakeup- triggered runs that produce >200 output tokens. Add a per-issue cooldown in enqueueWakeup(): if the target agent already had any completed run on this issue within the last 15 minutes, comment retrigger and comment mentioned wakeups are skipped. The check is fail-open (logs and proceeds on query error). Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…aperclipai#190) * feat(server): add browser evidence gates for code issue transitions (paperclipai#189) Add two new server-side gates that require agents to prove interactive browser testing before transitioning code issues (those with an executionWorkspaceId). Non-code issues and board users are exempt. Engineer evidence gate (in_review): requires browse command text + image attachment from the transitioning agent, scoped to the current review cycle via timestamp validation. QA browse evidence gate (done): requires browse command text + image attachment from the QA PASS author, preventing rubber-stamp QA passes based on static code inspection alone. Both gates use same-actor binding (evidence must come from the actor performing the transition) and timing validation (evidence must post-date the issue's last update). The regex pattern is explicitly labeled as an interim control — gameable but sufficient to enforce workflow discipline. Gate ordering: transition → delivery → engineer evidence → QA pass → QA browse evidence → comment-required. 14 new tests in browse-evidence-gate.test.ts, updated 7 existing test files to include listAttachments mock and browse evidence where needed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: guard evidence gate queries + decouple handoff test Address code review feedback: - Only fetch comments/attachments on in_review/done transitions (avoids 2 unnecessary DB queries on every other status change) - Decouple review-handoff auto-infer test from evidence gate by using non-code issue, keeping the test focused on @mention behavior Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
) Agents woken via @mention or issue assignment had no in-prompt context about which issue triggered the wake. The wake reason, issue ID, and comment ID were only available as environment variables, which weaker models (e.g. MiniMax M2.7) don't reliably check. This caused QA Agent runs to exit with "no inbox items" when the triggering issue didn't match their status-based search filters. Build context.paperclipWakeNote in heartbeat.ts (using already-fetched issueContext data) and inject it into all 6 adapter prompts via joinPromptSections(). For task-bound wakes, the note includes the issue identifier, title, wake reason, and triggering comment ID. For global heartbeats, it says to check the inbox. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…D-3596) Root cause: SPE agent stuck in error for 32 min (11:07-11:39 UTC) despite recoverProcessLostAgents sweeper running every ~2 min. Two failure modes identified: 1. Race with heartbeat transaction isolation: the heartbeat updates agents.status='error' in one transaction, then updates heartbeat_runs in a separate transaction. The sweeper reads the agent in error state but the heartbeat_runs update hasn't committed yet, so lastRun.errorCode is NULL and the agent is skipped. 2. Server restart with no retry queued: if the heartbeat's shouldRetry=false (e.g. server-wide restart, not per-process crash), no retry is enqueued. When the server restarts, reapOrphanedRuns finds the 'running' run stuck but may not have a process_lost error_code to trigger sweeper recovery. The agent stays in error with no evidence in heartbeat_runs. Fix: Two new defensive paths in recoverProcessLostAgents: - Stale running run path: if agent is error AND has a heartbeat_runs row with status='running' whose updatedAt is >5 min old (matching reapOrphanedRuns staleness threshold), treat it as a crashed heartbeat and force-recover. The heartbeat will pick it up on next tick and properly mark the run as failed/process_lost. - No-runs path: if agent is error with no heartbeat_runs rows AND agent.updatedAt is >5 min old, force-recover. This handles the case where the heartbeat crashed before recording the run (DB tx never committed) or shouldRetry=false left no evidence. Also reduced the process_lost delay from 60s to 30s to be more responsive while still avoiding race with in-flight retries (sweeper runs ~30s apart). Added 4 new test cases covering all three recovery paths. Closes DLD-3596.
…nts (DLD-3623) Skip unpicked_assignment_retrigger for routine_execution tasks that have not been picked up (originKind='routine_execution' AND startedAt=null). These are routine self-assigned tasks where the assignee will naturally pick them up on their next routine heartbeat. Retriggering causes phantom wake cascades where Monitor wakes every heartbeat on its own routine tasks and creates false escalation issues. Changes: - Added originKind, createdByAgentId, startedAt to candidate SELECT - Added skip condition: if originKind='routine_execution' AND startedAt=null, skip - Added 2 regression tests (skip unpicked routine task, retrigger picked routine task) - Extended seedIssueFixture to support originKind and startedAt overrides
… listLiveIssueByRoutineIds Adds a third fallback tier to both functions to catch stagnation/stale issues created by the Monitor that have no active heartbeat run attached. - findLiveExecutionIssue(): after context-snapshot-bound query returns null, query originKind=routine_execution + originId=routine.id with no heartbeat run join condition. - listLiveIssueByRoutineIds(): after context-snapshot fallback, run origin-only query for still-missing routine IDs. This fixes DLD-4090: Monitor stagnation monitor creating a new issue every cycle instead of coalescing into existing stagnation issues. Fixes: DLD-4099 Co-Authored-By: Paperclip <noreply@paperclip.ing>
…dd updatedAt to errorAgents select Fixes TS2300 (duplicate identifier) and TS2339 (updatedAt missing) that were blocking the Docker build for the stagnation monitor fix. Fixes: DLD-4099 Co-Authored-By: Paperclip <noreply@paperclip.ing>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e866356621
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+662
to
+666
| eq(issues.companyId, routine.companyId), | ||
| eq(issues.originKind, "routine_execution"), | ||
| eq(issues.originId, routine.id), | ||
| inArray(issues.status, OPEN_ISSUE_STATUSES), | ||
| isNull(issues.hiddenAt), |
There was a problem hiding this comment.
This fallback returns any open routine_execution issue even when no heartbeat run is queued/running, and runRoutine treats that as an active execution to coalesce/skip (findLiveExecutionIssue feeds the activeIssue check at lines 777-784). In practice, if a previous run completed but its issue is still open (for example left in todo/blocked), subsequent routine triggers will be incorrectly deduplicated instead of creating a new execution issue, which breaks the intended coalesce_if_active behavior and can leave routines stuck on stale issues.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a third fallback tier to both
findLiveExecutionIssue()andlistLiveIssueByRoutineIds()inserver/src/services/routines.ts.When a Monitor-created stagnation issue has no active
heartbeatRunattached, the two existing queries (execution-bound + context-snapshot-bound) both return null. The new origin-only query bridges the gap by matching onoriginKind=routine_execution+originId=routine.idalone, with no heartbeat run join.Also fixes pre-existing TypeScript build errors in
heartbeat.ts(duplicate function identifier + missingupdatedAtproperty).Fixes: DLD-4090
Closes: DLD-4099