Skip to content

fix(productivity-review): suppress reviews on dead assignees, exclude never-started runs - #8

Open
PraeSynBH wants to merge 2 commits into
masterfrom
rbr-983-productivity-monitor-dead-agent
Open

fix(productivity-review): suppress reviews on dead assignees, exclude never-started runs#8
PraeSynBH wants to merge 2 commits into
masterfrom
rbr-983-productivity-monitor-dead-agent

Conversation

@PraeSynBH

Copy link
Copy Markdown
Owner

Thinking Path

  • Paperclip is the open source app people use to manage AI agents for work
  • The productivity-review service watches issue-linked agent runs for signs an assignee is stuck or unresponsive, and mints a review issue when it sees one
  • It measured absence of output (no run-created comments, long elapsed active time) without checking whether the assignee agent actually had a live process to produce that output, and without excluding runs that were cancelled before they ever started executing
  • A dead agent (status: error, no process) cannot comment; a run cancelled by a reassignment race before it claimed the queue cannot comment either. Counting either as evidence of poor diligence inverts the signal — it blames the agent for an infrastructure fault
  • This pull request makes the monitor check assignee actability before firing, drops pre-start-cancelled runs from the no-comment streak, labels wall-clock-timeout-dominated streaks as a capacity signal rather than a productivity one, and avoids re-minting a review when a prior review on the same source issue already dispositioned the cause as non-productivity
  • The benefit is fewer false-positive productivity reviews against healthy agents caught in outages or scheduling races, and reviews that are minted read correctly as infra/capacity signals instead of performance complaints

Linked Issues or Issue Description

No public GitHub issue exists for this. Following the bug report template:

  • Current behavior: reconcileProductivityReviews counts every terminal run (including runs cancelled before they ever executed) toward the no-comment streak, and fires long_active_duration purely on elapsed wall-clock time — regardless of whether the assignee agent has a live process. An agent that goes into status: error (e.g. "Process lost — server may have restarted") produces a false-positive productivity review: the monitor reads "no comments" and "long elapsed time" as evidence of poor diligence when the agent had no process to act with.
  • Expected behavior: The monitor should suppress review generation entirely when the assignee cannot act (error/paused/terminated/pending_approval), exclude runs that never started executing (pre-start cancels from reassignment races) from the no-comment streak, label a streak dominated by wall-clock timeouts as a capacity signal rather than a productivity one, and avoid firing long_active_duration on an assignee that cannot act. If a prior review on the same source issue already closed with an infra/capacity disposition and the pattern recurs, it should extend that review rather than mint a duplicate.
  • Repro: seed an in_progress issue assigned to an agent with status: error, insert 9+ cancelled heartbeat runs with errorCode: issue_assignee_changed (no startedAt) and a 7h-elapsed active episode, run reconcileProductivityReviews. Before this change: fires both no_comment_streak and long_active_duration, minting a review that blames the agent. After: result.created === 0, zero review issues created, one issue.productivity_review_suppressed_infra activity log entry.

What Changed

  • resolveAssigneeActability: before evaluating any trigger, checks whether the assignee agent's lifecycle status (error/paused/terminated/pending_approval) means it currently has no live process. If so, the review is suppressed entirely and an issue.productivity_review_suppressed_infra activity log entry is written instead (rate-limited like refresh comments), routing the signal to the operator as an infra fault rather than a performance finding.
  • isPreStartCancelledRun / isExecutedRun: a run that terminated before the adapter ever started (identified by known pre-start-cancel error codes, a stopReason in the result payload, or the structural startedAt == null signal on a cancelled run) is excluded from the no-comment streak and from the executed-run counts used for evidence — only runs that actually executed are admissible evidence about the assignee's output.
  • classifySignal: labels the fired trigger's signalClass as productivity, capacity (wall-clock timeouts dominate the streak), or infra (assignee cannot act), and includes the classification and its reason in both the review body and the activity log so the review reads correctly.
  • long_active_duration no longer fires when the assignee cannot act — elapsed time against a dead/paused process measures an outage, not effort.
  • findUnresolvedDispositionedReviewCause: if the most recent done review on the same source issue was created with an infra or capacity signal class and the pattern recurs, extends that review with a recurrence comment (issue.productivity_review_recurrence_held) instead of minting a fresh review that re-derives the same conclusion.
  • Evidence fields and review body copy updated (executedRunCount, preStartCancelledRunCount, wallClockTimeoutRunCount, "No-comment executed-run streak") to reflect the corrected accounting.
  • New behavioral test: an agent in error with 9 no-comment pre-start-cancelled runs on an assigned in_progress issue (which would also independently trip long_active_duration) produces zero productivity reviews and exactly one suppression log entry.

Verification

Ran the full productivity-review service test file against a local embedded-postgres dev database:

cd server && npx vitest run src/__tests__/productivity-review-service.test.ts --no-coverage

Result: 12/12 passing, including the new AC6 behavioral test and an updated pre-existing assertion (No-comment completed-run streakNo-comment executed-run streak, matching the corrected copy). Per repo guidance on embedded-Postgres suite cost, did not run the repo-wide pnpm test or pnpm -r typecheck; ran a scoped npx tsc --noEmit pass over the server package and confirmed no new errors were introduced by this file (all pre-existing errors are unrelated plugin-sdk/plugin-host-services module-resolution issues, verified present before this change).

Risks

  • Low risk, additive logic gated behind new checks; no schema or API surface changes.
  • Behavioral shift: issues assigned to an agent in error/paused/terminated/pending_approval will no longer get productivity reviews at all while in that state — this is intentional (the whole point of the fix) but means a genuinely unproductive agent that later flips into error right before review time would also be suppressed until it recovers. Mitigated by the issue.productivity_review_suppressed_infra activity log entry, which still surfaces the situation to operators.
  • The pre-start-cancel error code allowlist (PRODUCTIVITY_REVIEW_PRE_START_CANCEL_ERROR_CODES) is a fixed set; a future cancel path with a new error code that reuses the pattern won't be recognized by name, but the structural startedAt == null fallback still catches it for any cancelled run.

Model Used

Claude (Anthropic), model claude-sonnet-5, via Claude Code CLI. Standard reasoning mode, with tool use (file read/search/edit, shell execution) to inspect the existing implementation, run the test suite, and fix a regressed assertion.

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I have searched GitHub for duplicate or related PRs and linked them above
  • I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
  • I have not referenced internal/instance-local Paperclip issues or links (only public GitHub #NNN / github.com/paperclipai/paperclip URLs)
  • My branch name describes the change (e.g. docs/..., fix/...) and contains no internal Paperclip ticket id or instance-derived details
  • I have run tests locally and they pass
  • I have added or updated tests where applicable
  • I have updated relevant documentation to reflect my changes
  • I have considered and documented any risks above
  • All Paperclip CI gates are green
  • Greptile is 5/5 with no open P2s, recommendations, or follow-ups
  • I will address all Greptile and reviewer comments before requesting merge

… never-started runs (RBR-983)

RBR-978 fired a no_comment_streak review on RBR-920 while the assignee
(CTO) was in `error` with no live process, and 9 of the 12 "completed"
runs never actually executed -- they cancelled each other in a 43s
reassignment race. The monitor measured the absence of output from a
dead process and attributed it to the agent's work habits.

- AC1: suppress review generation entirely when the assignee cannot
  act (`error`/`paused`/`terminated`/`pending_approval`); log an
  `issue.productivity_review_suppressed_infra` activity entry instead
  of minting a performance finding.
- AC2: exclude pre-start-cancelled runs (`issue_assignee_changed`,
  `lock_released_on_reassignment`, and other pre-start cancel codes,
  plus the structural `startedAt == null` signal) from the no-comment
  streak -- only executed runs are admissible evidence.
- AC3: classify a streak dominated by wall-clock timeouts as a
  `capacity` signal, not a `productivity` one, and say so explicitly
  in the review body.
- AC4: do not fire `long_active_duration` when the assignee cannot
  act -- elapsed time there measures outage, not effort.
- AC5: if the most recent dispositioned (`done`) review on the same
  source issue already closed with an infra/capacity signal class and
  the pattern recurs, extend that review with a recurrence comment
  instead of minting a fresh one.
- AC6: behavioral test -- an agent in `error` with 9 no-comment
  cancelled (never-started) runs on an assigned `in_progress` issue
  (which would also trip long_active_duration) produces zero
  productivity reviews and exactly one suppression log entry.

Verified: `vitest run src/__tests__/productivity-review-service.test.ts`
against a local embedded-postgres dev database -- 12/12 passing,
including the new AC6 test and the updated pre-existing streak-label
assertion. Per RBR-939, did not run repo-wide typecheck/pnpm test.
… beforeAll

Same embedded-postgres teardown cost as startup; afterAll was
flaking past vitest's 10s default under load while beforeAll already
carried 120_000. No behavioral change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants