Skip to content

fix(redis): preserve ioredis reconnect ownership - #2350

Merged
og2701 merged 4 commits into
devfrom
fix/dragonfly-resize-reconnect
Aug 4, 2026
Merged

fix(redis): preserve ioredis reconnect ownership#2350
og2701 merged 4 commits into
devfrom
fix/dragonfly-resize-reconnect

Conversation

@og2701

@og2701 og2701 commented Jul 22, 2026

Copy link
Copy Markdown
Member

Summary

  • keep ioredis in control of reconnecting after a provider-side connection close
  • use disconnect(true) only when repeated probe failures occur while the client still reports ready
  • add a socket-level regression test covering endpoint loss beyond the old stale threshold and subsequent recovery

Root cause

During an in-place Dragonfly resize, the provider can close an established connection. ioredis starts its normal reconnect loop, but the availability monitor previously treated a reconnect lasting more than five seconds as stale and called disconnect(false) followed by connect(). The explicit non-reconnecting disconnect could cancel the client-owned retry lifecycle and leave the process unable to recover when the endpoint returned.

Impact

Redis-backed features can recover automatically after Dragonfly closes connections during a resize, without restarting the application process.

Verification

  • Redis availability integration suite: 21 passed, 0 failed
  • focused reconnect regression using a real ioredis client and TCP endpoint: passed
  • server typecheck: passed
  • focused Biome check: passed
  • git diff --check: passed

A real staging Dragonfly in-place resize remains the final provider-level validation.


Summary by cubic

Keeps ioredis fully in charge of reconnects and isolates health checks on a duplicate client to prevent downtime during Dragonfly in-place resize. Also enables a staging deploy of this branch to validate recovery.

  • Bug Fixes

    • Remove stale reconnect logic; never force reconnects or call disconnect(false); let ioredis handle retries.
    • Run probes on redis.duplicate() created when monitoring starts; keep the request client untouched and warm both connections concurrently.
    • Only use Redis when it’s configured, redis.status === "ready", and the state is healthy; add regression tests for provider-close recovery and probe isolation.
  • CI

    • Add fix/dragonfly-resize-reconnect to STAGING_DEPLOY_BRANCH_ALLOWLIST to allow staging validation.

Written for commit 888961b. Summary will update on new commits.

Review in cubic

Greptile Summary

This PR preserves automatic Redis recovery when a provider closes an active connection.

  • [Bug fixes] Moves availability probes to an isolated duplicate Redis connection so monitoring cannot interrupt the request client's reconnect lifecycle.
  • [Bug fixes] Removes the stale-reconnect override and requires the request client to be ready before Redis-backed features are used.
  • [Improvements] Adds socket-level regression coverage for endpoint loss and recovery.
  • [Improvements] Allows the fix branch to deploy to staging for provider-level validation.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
server/src/external/redis/initUtils/createRedisAvailability.ts Isolates health probes on a duplicate connection, preserves request-client reconnect ownership, and cleans up monitor resources.
server/tests/integration/others/redis/create-redis-availability.test.ts Adds unit and socket-level regression coverage for probe isolation, concurrent warmup, endpoint loss, and recovery.
.github/workflows/build.yml Adds the PR branch to the temporary staging-deployment allowlist.

Sequence Diagram

sequenceDiagram
    participant Monitor as Availability monitor
    participant Probe as Probe Redis client
    participant Request as Request Redis client
    participant Endpoint as Redis endpoint

    Monitor->>Probe: PING
    Probe->>Endpoint: PING
    Endpoint-->>Probe: PONG
    Endpoint--xRequest: Provider closes connection
    Request->>Request: ioredis owns retry lifecycle
    Monitor->>Probe: Continue availability probes
    Endpoint-->>Request: Endpoint becomes available
    Request->>Endpoint: Reconnect automatically
    Request-->>Monitor: Status ready
Loading

Reviews (2): Last reviewed commit: "Merge existing PR #2350 history" | Re-trigger Greptile

@og2701
og2701 marked this pull request as ready for review July 22, 2026 12:07
@og2701
og2701 requested review from ay-rod and johnyeocx as code owners July 22, 2026 12:07
@vercel

vercel Bot commented Jul 22, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments
Project Deployment Actions Updated (UTC)
checkout Ignored Ignored Jul 22, 2026 12:07pm
landing-page Ignored Ignored Jul 22, 2026 12:07pm

Request Review

@capy-ai

capy-ai Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Capy auto-review is paused for this organization because the usage-cycle auto-review limit has been reached. Increase the limit or turn it off in billing settings to resume automatic reviews.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 3 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

@og2701 og2701 closed this Jul 28, 2026
@og2701 og2701 reopened this Aug 4, 2026
@og2701
og2701 merged commit 6cc5214 into dev Aug 4, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant