Skip to content

[BUGFIX] Ring: fix DoBatch cleanup callback blocked forever on callback panic - #7559

Merged
SungJin1212 merged 3 commits into
cortexproject:masterfrom
sandy2008:fix/distributor-dobatch-context-leak
May 29, 2026
Merged

[BUGFIX] Ring: fix DoBatch cleanup callback blocked forever on callback panic#7559
SungJin1212 merged 3 commits into
cortexproject:masterfrom
sandy2008:fix/distributor-dobatch-context-leak

Conversation

@sandy2008

@sandy2008 sandy2008 commented May 27, 2026

Copy link
Copy Markdown
Contributor

What this PR does

ring.DoBatch submitted closures called wg.Done() inline after callback() and tracker.record(). If callback panicked, wg.Done() was skipped, causing wg.Wait() to block forever and the cleanup callback to never execute.

Symptom: any panic inside a DoBatch callback permanently blocked the cleanup goroutine. For the distributor, this leaked the context.WithTimeout timer (until RemoteTimeout expired) and request buffers (req.Timeseries, req.Free()) — they were never reclaimed. The same issue affected all three production DoBatch callers: distributor, alertmanager distributor, and multitenant alertmanager.

Fix: move wg.Done() to defer wg.Done() so it runs even during panic unwinding, ensuring the cleanup goroutine always completes. An earlier iteration also added defer cancel() to the distributor's doBatch function, but this was intentionally reverted after review found it contradicts the design intent: "Use a background context to make sure all ingesters get samples even if we return early." The defer wg.Done() fix makes the existing cleanup callback (which already calls cancel()) reliable on all paths, preserving send semantics without cancelling in-flight ingester RPCs.

Which issue(s) this PR fixes

Fixes #7558

Checklist

  • CHANGELOG.md updated — not yet, will add if maintainers confirm the fix direction.
  • Documentation updated — not applicable, no flags or config changed.
  • Tests added — TestDoBatchCleanupCalledOnCallbackPanic verifies cleanup runs even when callback panics.

Test plan

  • go vet ./pkg/ring/... ./pkg/distributor/... — clean
  • go test -tags "netgo slicelabels" -count=1 -timeout 120s ./pkg/ring/... — 10/10 packages pass
  • go test -tags "netgo slicelabels" -run TestDistributor_BatchTimeoutMetric ./pkg/distributor/... — passes
  • New test passes with fix, fails without (verified by reverting defer during development)

DoBatch's submitted closures called wg.Done() inline after callback()
and tracker.record(). If callback panicked, wg.Done() was skipped,
causing wg.Wait() to block forever and the cleanup callback to never
execute. This leaked context timers, request buffers, and any other
resources owned by the cleanup function for all DoBatch callers
(distributor, alertmanager).

Move wg.Done() to a defer so it runs even during panic unwinding,
ensuring the cleanup goroutine always completes.

Fixes cortexproject#7558

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Sandy Chen <Yuxuan.Chen@morganstanley.com>
@dosubot dosubot Bot added component/ring go Pull requests that update Go code type/bug labels May 27, 2026
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Sandy Chen <Yuxuan.Chen@morganstanley.com>
@sandy2008 sandy2008 changed the title Fix DoBatch cleanup callback blocked forever on callback panic [BUGFIX] Ring: fix DoBatch cleanup callback blocked forever on callback panic May 27, 2026
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Sandy Chen <Yuxuan.Chen@morganstanley.com>

@friedrichg friedrichg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. the test proves to me this is needed

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label May 29, 2026
@SungJin1212
SungJin1212 merged commit 9fb947d into cortexproject:master May 29, 2026
37 checks passed
CharlieTLe added a commit to CharlieTLe/cortex that referenced this pull request Sep 2, 2026
Get the unreleased section into the shape RELEASE.md asks for before the
release-1.22 cut, so operators reading the notes see the changes that affect them:

- Delete the stale duplicate of cortexproject#7375, which already shipped in 1.21.0 as cortexproject#7370.
- Re-sort into CHANGE -> FEATURE -> ENHANCEMENT -> BUGFIX.
- Correct the gRPC entry: the bump landed at v1.82.1, not v1.79.3.
- Fold follow-up PRs into the entry they belong to.
- Add three missing user-facing entries: cortexproject#7513, cortexproject#7514 and cortexproject#7559.
- Reclassify as CHANGE the entries that break existing configs or log consumers:
  the sign-key validation (cortexproject#7587), the Alertmanager per-tenant *_file rejections
  (cortexproject#7767, cortexproject#7768, now one entry) and the time_taken -> time_taken_ms rename (cortexproject#7649).
- Note the operator impact of the distroless base image (cortexproject#7637) and of the
  500 -> 499 reclassification (cortexproject#7717).

Rebased onto master, which added eight entries after this was first written.
They are curated the same way:

- Sorted into their categories: the deprecated flag removal (cortexproject#7790) and the
  max-exemplars deprecation (cortexproject#7793) under CHANGE, the X-Grafana-User query log
  (cortexproject#7799) under ENHANCEMENT, the YAML zero-value validation (cortexproject#7700) and the
  ReadPartitionedGroupInfo error handling (cortexproject#7766) under BUGFIX.
- Folded the Go toolchain bump (cortexproject#7807, cortexproject#7814) into the existing build image
  entry, which now reads 1.27.0 rather than carrying a second entry for it.
- Folded cortexproject#7745 into cortexproject#7698: both are the same wipe-on-transient-DNS-failure bug,
  cortexproject#7698 on the A record path and cortexproject#7745 on the SRV path.
- Folded cortexproject#7743 into cortexproject#7640: both are panics in the active request tracker's
  truncation of match[]/query values.

Signed-off-by: Charlie Le <charlie_le@apple.com>
CharlieTLe added a commit to CharlieTLe/cortex that referenced this pull request Sep 11, 2026
Get the unreleased section into the shape RELEASE.md asks for before the
release-1.22 cut, so operators reading the notes see the changes that affect them:

- Delete the stale duplicate of cortexproject#7375, which already shipped in 1.21.0 as cortexproject#7370.
- Re-sort into CHANGE -> FEATURE -> ENHANCEMENT -> BUGFIX.
- Correct the gRPC entry: the bump landed at v1.82.1, not v1.79.3.
- Fold follow-up PRs into the entry they belong to.
- Add three missing user-facing entries: cortexproject#7513, cortexproject#7514 and cortexproject#7559.
- Reclassify as CHANGE the entries that break existing configs or log consumers:
  the sign-key validation (cortexproject#7587), the Alertmanager per-tenant *_file rejections
  (cortexproject#7767, cortexproject#7768, now one entry) and the time_taken -> time_taken_ms rename (cortexproject#7649).
- Note the operator impact of the distroless base image (cortexproject#7637) and of the
  500 -> 499 reclassification (cortexproject#7717).

Rebased onto master, which added eight entries after this was first written.
They are curated the same way:

- Sorted into their categories: the deprecated flag removal (cortexproject#7790) and the
  max-exemplars deprecation (cortexproject#7793) under CHANGE, the X-Grafana-User query log
  (cortexproject#7799) under ENHANCEMENT, the YAML zero-value validation (cortexproject#7700) and the
  ReadPartitionedGroupInfo error handling (cortexproject#7766) under BUGFIX.
- Folded the Go toolchain bump (cortexproject#7807, cortexproject#7814) into the existing build image
  entry, which now reads 1.27.0 rather than carrying a second entry for it.
- Folded cortexproject#7745 into cortexproject#7698: both are the same wipe-on-transient-DNS-failure bug,
  cortexproject#7698 on the A record path and cortexproject#7745 on the SRV path.
- Folded cortexproject#7743 into cortexproject#7640: both are panics in the active request tracker's
  truncation of match[]/query values.

Rebased again onto master, which added eight more entries. Same treatment:

- Sorted into their categories: the evaluation-delay-duration removal (cortexproject#7792)
  and the fifocache/ingester-metadata-streaming removals (cortexproject#7791) under CHANGE,
  the parquet max-block-label-names limit (cortexproject#7625), the non-pointer
  HistogramBucket slice (cortexproject#7809) and the merge iterator BatchSize (cortexproject#7823) under
  ENHANCEMENT, and the CSV-list empty-string fix (cortexproject#7714) under BUGFIX.
- Folded the Thanos/promql-engine refresh (cortexproject#7788) into the existing upgrade
  entry, which already carries cortexproject#7691, cortexproject#7505 and cortexproject#7740.

Signed-off-by: Charlie Le <charlie_le@apple.com>
CharlieTLe added a commit that referenced this pull request Sep 12, 2026
* Pull minio from quay.io instead of Docker Hub

Every integration leg is failing at Preload Images:

  Error response from daemon: pull access denied for minio/minio,
  repository does not exist or may require 'docker login':
  denied: requested access to the resource is denied

minio/minio is the first Docker Hub pull in the step, so no leg gets past
it and all 24 fail in about 30 seconds. A CHANGELOG-only pull request
reproduces it, so this is not specific to any change under test. master
was last green at 4061a3d.

This is not a rate limit: the same pull fails right after a successful
'docker login' with the repository credentials. The docker.io/minio/minio
repository is simply no longer accessible.

MinIO still publishes the identical image to quay.io. quay.io/minio/minio
:RELEASE.2024-05-28T17-19-04Z is public and is a manifest list with 8
children, so it covers both the amd64 and arm64 runners.

Point the integration tests, the CI preload list, and the three
development docker-compose stacks at quay.io. The tag is unchanged, so no
behaviour changes.

Signed-off-by: Charlie Le <charlie_le@apple.com>

* Authenticate the integration job's remaining Docker Hub pulls

Preload Images still pulls consul, memcached, redis and postgres from
Docker Hub. #7464 removed the Install Docker Client step from this job,
and that script is where 'docker login' runs, so those pulls have been
anonymous since and are subject to the anonymous rate limit.

Log in explicitly, matching what the build job does. Pull requests from
forks have no secrets, so skip the login there and leave those pulls
anonymous instead of failing the step.

This is hardening, not the fix for the current breakage: minio failed
even when authenticated.

Signed-off-by: Charlie Le <charlie_le@apple.com>

* Resolve the latest release image to a published version

Backport of #7786 onto release-1.22.

integration/util.go derived the query fuzz comparison image straight from
VERSION. The moment VERSION becomes 1.22.0-rc.0 on this branch,
integration_query_fuzz tries to pull
quay.io/cortexproject/cortex:v1.22.0-rc.0, which does not exist: that
image is pushed by the tag build's deploy job, and deploy is gated on
integration passing first.

Ask quay.io which GA tags are actually published instead, and take the
highest one at or below VERSION. The CI step mirrors the same resolution
and exports CORTEX_LATEST_RELEASE_IMAGE for the preload step.

Set the CORTEX_LATEST_RELEASE_IMAGE repository variable to bypass the
lookup.

Signed-off-by: Charlie Le <charlie_le@apple.com>

* Mark release 1.22.0 in progress

Add a '## 1.22.0 in progress' section below an empty
'## master / unreleased', move the existing unreleased entries into it,
and order them [CHANGE] -> [FEATURE] -> [ENHANCEMENT] -> [BUGFIX] per
RELEASE.md.

Also fill the gaps reported by
./tools/release/check-changelog.sh v1.21.1...master:

- new entries for #7513, #7514 and #7559
- fold #7323, #7434, #7458, #7463, #7487, #7505, #7687, #7691, #7716,
  #7726, #7775 and #7807 into the entries they belong to

Signed-off-by: Charlie Le <charlie_le@apple.com>

* Update version to 1.22.0-rc.0

Signed-off-by: Charlie Le <charlie_le@apple.com>

---------

Signed-off-by: Charlie Le <charlie_le@apple.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/ring go Pull requests that update Go code lgtm This PR has been approved by a maintainer size/L type/bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Distributor: context.WithTimeout timer leak in doBatch — cancel() not deferred, relies solely on ring.DoBatch cleanup callback

3 participants