Skip to content

[exporter/prometheus_remote_writeexporter] Fix WAL buffered data stall … - #49131

Merged
ChrsMark merged 2 commits into
open-telemetry:mainfrom
charanck9:fix/prw-wal-buffered-data-stall
Aug 5, 2026
Merged

[exporter/prometheus_remote_writeexporter] Fix WAL buffered data stall …#49131
ChrsMark merged 2 commits into
open-telemetry:mainfrom
charanck9:fix/prw-wal-buffered-data-stall

Conversation

@charanck9

Copy link
Copy Markdown
Contributor

…on idle

Add a timeout to the WAL read wait so that buffered entries are flushed even when no new data arrives. Previously, readPrompbFromWAL would block indefinitely on rNotify when the WAL was empty, preventing the truncation timer in continuallyPopWALThenExport from ever firing. This caused buffered requests to remain unsent until the next write.

Also add the exporter ID to WAL telemetry attributes for disambiguation when multiple exporters share a collector.

Description

Link to tracking issue

#49130
Fixes

Testing

Documentation

Authorship

  • [yes ] I, a human, wrote this pull request description myself.

@charanck9
charanck9 requested review from a team, ArthurSens and dashpole as code owners June 17, 2026 07:10
@linux-foundation-easycla

linux-foundation-easycla Bot commented Jun 17, 2026

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

@github-actions github-actions Bot added the first-time contributor PRs made by new contributors label Jun 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Welcome, contributor! Thank you for your contribution to opentelemetry-collector-contrib.

Important reminders:

  • Read our Contributing Guidelines.
  • Sign the CLA if you haven't already.
  • First-time contributors should have at most one PR not marked as draft until their first PR is merged.
  • If your change isn't one of our priority components, reviews may take more time.
  • Give reviewers at least a few days before pinging them for feedback.
  • If you need help or struggle to move your PR forward:

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses an idle-stall scenario in the Prometheus Remote Write exporter WAL reader by ensuring the export/truncation loop can make progress even when no new WAL entries arrive, and improves WAL telemetry by adding exporter identity attributes to disambiguate metrics across multiple exporter instances.

Changes:

  • Add an exporter attribute to WAL telemetry metrics for per-exporter disambiguation.
  • Introduce a bounded wait (timeout) when the WAL read hits ErrNotFound, allowing the export loop to periodically re-check truncation/export conditions.
  • On read timeout, opportunistically flush buffered requests when the truncation timer has fired.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread exporter/prometheusremotewriteexporter/wal.go
Comment thread exporter/prometheusremotewriteexporter/wal.go
@charanck9
charanck9 force-pushed the fix/prw-wal-buffered-data-stall branch from 6a2aa4a to ed8cea8 Compare June 30, 2026 13:26

@charanck9 charanck9 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

Comment thread exporter/prometheusremotewriteexporter/wal.go
Comment thread exporter/prometheusremotewriteexporter/wal.go

@charanck9 charanck9 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resolved the comments

@github-actions

Copy link
Copy Markdown
Contributor

This PR was marked stale due to lack of activity. It will be closed in 14 days.

@github-actions github-actions Bot added the Stale label Jul 16, 2026
@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Jul 18, 2026

Copy link
Copy Markdown

Pull request dashboard status

Merged · refreshed 2026-08-05 06:39 UTC

Status above doesn't look right?
  • Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

@krajorama krajorama left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix is looking good, but please add chloggen entry and also split out the unrelated attribute change. Smaller PRs are faster to approve and merge.

Comment thread exporter/prometheusremotewriteexporter/wal.go Outdated

@charanck9 charanck9 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments addressed

@charanck9 charanck9 changed the title [exporter/prometheusremotewriteexporter] Fix WAL buffered data stall … [exporter/prometheus_remote_writeexporter] Fix WAL buffered data stall … Jul 29, 2026
@charanck9
charanck9 force-pushed the fix/prw-wal-buffered-data-stall branch 2 times, most recently from 5b26578 to 5ba18fc Compare July 29, 2026 09:12
Signed-off-by: charanck9 <charan.ulasala.ext@ericsson.com>
@charanck9
charanck9 force-pushed the fix/prw-wal-buffered-data-stall branch from 5ba18fc to 7eff7e3 Compare July 29, 2026 09:47
@github-actions github-actions Bot removed the Stale label Jul 30, 2026
@opentelemetry-pr-dashboard

Copy link
Copy Markdown

Hi @charanck9 — just a friendly reminder that this pull request is waiting on you.

There are still items that need your attention. See the dashboard status comment for the full list. You don't need to push a code change to hand it back — replying to move each discussion forward is enough, whether that's answering a question, explaining why no change is needed, or asking a follow-up. The dashboard then automatically routes it back to reviewers.

If you believe this pull request is incorrectly routed as waiting on the author, comment /dashboard route:reviewers to route it from waiting on the author to waiting on reviewers.

@singhvibhanshu

Copy link
Copy Markdown
Member

/workflow-approve

@singhvibhanshu

Copy link
Copy Markdown
Member

/rerun

@singhvibhanshu singhvibhanshu added the ready to merge Code review completed; ready to merge by maintainers label Aug 5, 2026
@singhvibhanshu

Copy link
Copy Markdown
Member

/workflow-approve

@ChrsMark
ChrsMark merged commit 01c0052 into open-telemetry:main Aug 5, 2026
214 of 216 checks passed
@otelbot

otelbot Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution @charanck9! 🎉 We would like to hear from you about your experience contributing to OpenTelemetry by taking a few minutes to fill out this survey.

@charanck9
charanck9 deleted the fix/prw-wal-buffered-data-stall branch August 5, 2026 06:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

exporter/prometheusremotewrite first-time contributor PRs made by new contributors ready to merge Code review completed; ready to merge by maintainers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants