fix(server): throttle and back off heartbeat scheduler errors (#10911)

[wakqasahmed]

Fixes #10911

Summary

When PostgreSQL becomes unreachable or fails while Paperclip server is running, the background heartbeat scheduler ticks (heartbeat.tickTimers, routines.tickScheduledTriggers, sweepers) fail continuously on every interval tick and log without throttling. This resulted in writing gigabytes of logs per minute (e.g. 5.35 GB in 130 seconds).

Fix

1. Error Throttling & Deduplication: Added ErrorThrottler to deduplicate and rate-limit repeated identical background error messages within a 5-minute suppression window. The first occurrence logs full details, while subsequent identical errors are suppressed and emitted as a periodic summary line with the count of suppressed occurrences. Truncates long SQL strings to keep individual log lines compact.
2. Adaptive Scheduler Backoff: Implemented exponential backoff for the heartbeat scheduler interval upon consecutive tick failures (doubling up to a 5-minute max cap), resetting to normal (30s) on successful ticks.
3. Unit & Integration Tests: Added test coverage in server/src/__tests__/error-throttler.test.ts and server/src/__tests__/heartbeat-scheduler-throttling.test.ts.

[commitperclip]

Hey @wakqasahmed! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[greptile-apps]

Greptile Summary

The PR adds deduplicated scheduler error logging and replaces the fixed heartbeat interval with adaptive timeout-based scheduling. It also adds unit tests for throttling and the intended backoff sequence.

• Adds ErrorThrottler with message truncation, per-error suppression state, and periodic summaries.
• Routes scheduler and sweeper errors through the shared throttler.
• Adds exponential scheduling delays after failures and resets the delay after recovery.
• Adds throttling and backoff calculation tests.

Confidence Score: 4/5

The scheduler backoff should be corrected before merging because most background-task failures never reach its failure counter.

Individual scheduler jobs consume their own rejections, while only rejection of the outer callback increments the new consecutive-failure state, leaving affected jobs at the normal cadence.

Files Needing Attention: server/src/index.ts

Important Files Changed

Filename	Overview
server/src/index.ts	Integrates throttling and adaptive scheduling, but locally handled task failures do not update the backoff counter.
server/src/lib/error-throttler.ts	Adds error summarization, keyed suppression, periodic suppressed-count logging, and reset operations.
server/src/tests/error-throttler.test.ts	Covers summarization, suppression, summary emission, distinct keys, and reset behavior.
server/src/tests/heartbeat-scheduler-throttling.test.ts	Demonstrates throttler behavior and duplicates the backoff formula, but does not exercise the production scheduler control flow where failures are consumed.

Prompt To Fix All With AI

### Issue 1
server/src/index.ts:949-955
**Task failures bypass scheduler backoff**

When scheduled jobs such as `tickTimers` or `tickScheduledTriggers` reject, their local catch handlers consume the errors, so `await callback()` resolves without incrementing `consecutiveSchedulerFailures`. The scheduler therefore continues at its base interval instead of applying exponential backoff while those jobs repeatedly fail.

### Issue 2
server/src/__tests__/heartbeat-scheduler-throttling.test.ts:5
**PR description omits required sections**

The PR description does not include the required Thinking Path, Verification, Risks, or Model Used sections. Please add the complete template information so maintainers can evaluate how the scheduler change was verified and what operational risks remain.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "fix(server): throttle and back off heart..." | Re-trigger Greptile}

[greptile-apps]

Task failures bypass scheduler backoff

When scheduled jobs such as tickTimers or tickScheduledTriggers reject, their local catch handlers consume the errors, so await callback() resolves without incrementing consecutiveSchedulerFailures. The scheduler therefore continues at its base interval instead of applying exponential backoff while those jobs repeatedly fail.

Knowledge Base Used: Server HTTP App

Prompt To Fix With AI

This is a comment left during a code review.
Path: server/src/index.ts
Line: 949-955

Comment:
**Task failures bypass scheduler backoff**

When scheduled jobs such as `tickTimers` or `tickScheduledTriggers` reject, their local catch handlers consume the errors, so `await callback()` resolves without incrementing `consecutiveSchedulerFailures`. The scheduler therefore continues at its base interval instead of applying exponential backoff while those jobs repeatedly fail.

**Knowledge Base Used:** [Server HTTP App](https://app.greptile.com/paperclip-org-3/-/custom-context/knowledge-base/paperclipai/paperclip/-/docs/server-http-app.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

PR description omits required sections

The PR description does not include the required Thinking Path, Verification, Risks, or Model Used sections. Please add the complete template information so maintainers can evaluate how the scheduler change was verified and what operational risks remain.

Context Used: CONTRIBUTING.md has a guide for a good PR message ... (source)

Prompt To Fix With AI

This is a comment left during a code review.
Path: server/src/__tests__/heartbeat-scheduler-throttling.test.ts
Line: 5

Comment:
**PR description omits required sections**

The PR description does not include the required Thinking Path, Verification, Risks, or Model Used sections. Please add the complete template information so maintainers can evaluate how the scheduler change was verified and what operational risks remain.

**Context Used:** CONTRIBUTING.md has a guide for a good PR message ... ([source](https://app.greptile.com/review/custom-context?memory=instruction-0))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!