Skip to content

fix: shard async track FIFO message groups - #2357

Merged
SirTenzin merged 1 commit into
mainfrom
fix/async-track-fifo-sharding
Jul 22, 2026
Merged

fix: shard async track FIFO message groups#2357
SirTenzin merged 1 commit into
mainfrom
fix/async-track-fifo-sharding

Conversation

@og2701

@og2701 og2701 commented Jul 22, 2026

Copy link
Copy Markdown
Member

Shards async track FIFO message groups deterministically across eight lanes using the SQS deduplication ID. This applies only to the dedicated async track queue; the fail-open track queue keeps its existing customer/entity grouping.\n\nTests: server typecheck; 16 targeted async track, batch track, and queue track unit tests.


Summary by cubic

Shard async track SQS FIFO message groups into 8 deterministic lanes using the message deduplication ID to reduce head-of-line blocking and improve throughput. Applies only to the dedicated async track queue; the fail-open queue keeps its existing customer/entity grouping.

  • Bug Fixes
    • Added getAsyncTrackMessageGroupId to hash the dedupe ID into 8 shards; group ID format: org:env:customer:entity:shard-[0-7].
    • Updated runAsyncTrack and runBatchTrack to set the new messageGroupId; queueTrack now accepts an optional messageGroupId.
    • Expanded unit tests to verify deterministic sharding and coverage across all 8 shards.

Written for commit 29d6160. Summary will update on new commits.

Review in cubic

Greptile Summary

This PR shards messages on the dedicated async track FIFO queue across eight lanes. The main changes are:

  • Improvements: Derive an async message group from the SQS deduplication ID.
  • Improvements: Apply the new grouping to single and batch async track requests.
  • Bug fixes: Preserve customer and entity grouping for the separate fail-open queue.
  • Improvements: Add tests for shard format, determinism, and distribution.

Confidence Score: 4/5

The async grouping path can reorder balance-affecting events for the same customer and entity.

  • Request-specific shard keys split one ordering domain across independent FIFO groups.
  • Single requests and entries within one batch can execute in a different order from enqueue order.
  • The fail-open queue retains its previous grouping behavior.

server/src/internal/balances/track/utils/getAsyncTrackMessageGroupId.ts

Important Files Changed

Filename Overview
server/src/internal/balances/track/utils/getAsyncTrackMessageGroupId.ts Adds eight-lane hashing based on each message ID, which removes customer and entity FIFO serialization.
server/src/internal/balances/track/runAsyncTrack.ts Routes each dedicated async request using its request ID as the deduplication and shard input.
server/src/internal/balances/track/runBatchTrack.ts Assigns each batch entry a separately hashed lane, allowing entries for one customer and entity to run out of array order.
server/src/internal/balances/track/utils/queueTrack.ts Adds an optional message-group override while retaining the existing fallback for fail-open callers.
server/tests/unit/balances/track/runAsyncTrack.test.ts Updates single-request tests to cover the sharded group format and deterministic hashing.
server/tests/unit/balances/track/runBatchTrack.test.ts Updates batch tests to cover shard formatting and distribution across all eight lanes.

Sequence Diagram

sequenceDiagram
    participant Client
    participant API
    participant A as FIFO shard A
    participant B as FIFO shard B
    participant Worker
    Client->>API: Event 1 for customer/entity
    API->>A: Group from request ID 1
    Client->>API: Event 2 for same customer/entity
    API->>B: Group from request ID 2
    B->>Worker: Event 2 may run first
    A->>Worker: Event 1 may run later
    Note over Worker: Balance operations observe reversed order
Loading
Prompt To Fix All With AI
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
server/src/internal/balances/track/utils/getAsyncTrackMessageGroupId.ts:18-19
**Customer Events Lose FIFO Order**

The shard depends on the per-message deduplication ID, so sequential events for the same customer and entity usually enter different SQS message groups. Workers can then process later requests—or later entries from one batch—before earlier ones, causing order-sensitive balance caps, rejections, and shared-balance deductions to observe the wrong state; the previous group key serialized these events.

Reviews (1): Last reviewed commit: "🐛 fix(track): shard async FIFO message ..." | Re-trigger Greptile

Greptile also left 1 inline comment on this PR.

Context used:

  • Context used - When generating the key changes section of the sum... (source)

@og2701
og2701 requested review from ay-rod and johnyeocx as code owners July 22, 2026 16:11
@capy-ai

capy-ai Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Capy auto-review is paused for this organization because the usage-cycle auto-review limit has been reached. Increase the limit or turn it off in billing settings to resume automatic reviews.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 6 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

@SirTenzin
SirTenzin merged commit 2d102ea into main Jul 22, 2026
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants