Skip to content

[extension/opampextension] Fix self-reported status events dropped by non-Reporter hosts - #49412

Open
CodeBlackwell wants to merge 5 commits into
open-telemetry:mainfrom
CodeBlackwell:crg-farm/opamp-agent-fix
Open

[extension/opampextension] Fix self-reported status events dropped by non-Reporter hosts#49412
CodeBlackwell wants to merge 5 commits into
open-telemetry:mainfrom
CodeBlackwell:crg-farm/opamp-agent-fix

Conversation

@CodeBlackwell

Copy link
Copy Markdown

Root cause

opampAgent.Start wires the extension's internal reportFunc (used to report the extension's own status, e.g. from monitorPPID) directly to componentstatus.ReportStatus(host, event).

componentstatus.ReportStatus is a no-op when the component.Host passed to Start does not implement componentstatus.Reporter. In that case, self-reported events from the opamp extension — most notably the fatal error monitorPPID emits when the collector's parent process disappears — were silently discarded instead of reaching the extension's own status aggregator. The collector could keep reporting healthy even though it had been orphaned.

The fix

  • Added an extensionID component.ID field to opampAgent, populated from extension.Settings.ID in newOpampAgent.
  • In Start, build selfInstanceID := componentstatus.NewInstanceID(o.extensionID, component.KindExtension) and route reportFunc through the extension's own ComponentStatusChanged handler (o.ComponentStatusChanged(selfInstanceID, event)) instead of componentstatus.ReportStatus(host, event).

ComponentStatusChanged always forwards events into the extension's internal status aggregator, so self-reported events now reach it regardless of whether the host implements componentstatus.Reporter.

Before / after

  • Before: self-reported status events (fatal error on parent-process disappearance, etc.) were dropped whenever host didn't implement componentstatus.Reporter (e.g. componenttest.NewNopHost()), so the collector's health state never reflected them.
  • After: self-reported events are delivered to the extension's own status aggregator unconditionally, independent of the host's capabilities.

Test added

TestReportFuncReachesStatusAggregatorRegardlessOfHostReporter (extension/opampextension/opamp_agent_test.go) starts the agent against componenttest.NewNopHost() — which intentionally does not implement componentstatus.Reporter — reports a fatal error via o.reportFunc, and asserts the mock status aggregator receives it with StatusFatalError and the original error.

Status

Fix was produced and verified by an automated CRG (code-review-graph) debug pass: discovery → adversarial verification → TDD fix (RED before the fix, GREEN after) → final gate. Final gate: clean.

Draft PR — opening for early feedback; not requesting review yet.

@linux-foundation-easycla

linux-foundation-easycla Bot commented Jul 1, 2026

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

@github-actions github-actions Bot added the first-time contributor PRs made by new contributors label Jul 1, 2026
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Welcome, contributor! Thank you for your contribution to opentelemetry-collector-contrib.

Important reminders:

  • Read our Contributing Guidelines.
  • Sign the CLA if you haven't already.
  • First-time contributors should have at most one PR not marked as draft until their first PR is merged.
  • If your change isn't one of our priority components, reviews may take more time.
  • Give reviewers at least a few days before pinging them for feedback.
  • If you need help or struggle to move your PR forward:

@CodeBlackwell
CodeBlackwell marked this pull request as ready for review July 1, 2026 03:14

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: eb2668a22f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

selfInstanceID := componentstatus.NewInstanceID(o.extensionID, component.KindExtension)
o.reportFunc = func(event *componentstatus.Event) {
componentstatus.ReportStatus(host, event)
o.ComponentStatusChanged(selfInstanceID, event)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Route fatal self-reports through the host reporter

With the normal collector service host, host implements componentstatus.Reporter; before this change the self-report went through that reporter, whose status notification path also sends StatusFatalError to the service AsyncErrorChannel. Calling the OpAMP watcher directly only updates this extension's internal aggregator, so when ppid is configured and the parent disappears the OpAMP health may flip but the collector no longer receives the fatal event and keeps running orphaned (and with reports_health: false this send has no initialized componentStatusCh to receive it). Please preserve the host reporter path when available and add the internal fallback for non-reporter hosts.

Useful? React with 👍 / 👎.

Comment thread extension/opampextension/opamp_agent.go
…ped by non-Reporter hosts

opampAgent.Start wired its internal reportFunc to call
componentstatus.ReportStatus(host, event) directly. That helper silently
no-ops when the component.Host passed to Start does not implement
componentstatus.Reporter, so self-reported status changes -- most notably
the fatal error monitorPPID emits when the collector's parent process
disappears -- never reached the extension's own status aggregator. The
collector could keep reporting healthy despite being orphaned.

Track the extension's own component.ID (added extensionID field, set from
extension.Settings.ID in newOpampAgent) and route self-reported events
through the extension's own ComponentStatusChanged handler instead, which
always forwards to the status aggregator regardless of what the host
implements.

Adds TestReportFuncReachesStatusAggregatorRegardlessOfHostReporter, which
starts the agent against componenttest.NewNopHost() (which intentionally
does not implement componentstatus.Reporter) and asserts a fatal error
event reported via o.reportFunc still reaches the mock status aggregator.
@CodeBlackwell
CodeBlackwell force-pushed the crg-farm/opamp-agent-fix branch from eb2668a to 42a171e Compare July 1, 2026 16:31
Per reviewer feedback (@avy252004, chatgpt-codex-connector): the initial fix
replaced componentstatus.ReportStatus(host, event) outright with the internal
ComponentStatusChanged handler. That fixed the non-Reporter (NopHost) case but
regressed production: with the real collector service host, ReportStatus routes
through the status notification path, which also forwards a StatusFatalError to
the service's AsyncErrorChannel so an orphaned collector actually shuts down.
Bypassing it left the collector running orphaned.

reportFunc now prefers the host reporter when the host implements
componentstatus.Reporter, and only falls back to the extension's own
ComponentStatusChanged handler for non-Reporter hosts, where ReportStatus would
otherwise be a silent no-op.

Adds TestReportFuncPrefersHostReporter covering the reporter path; the existing
TestReportFuncReachesStatusAggregatorRegardlessOfHostReporter still covers the
non-Reporter fallback.
@CodeBlackwell

Copy link
Copy Markdown
Author

Thanks @avy252004 and @chatgpt-codex-connector — you're both right. The first version replaced componentstatus.ReportStatus(host, event) outright, which fixed the non-Reporter (NopHost) case but regressed production: with the real service host, ReportStatus also forwards a StatusFatalError to the service's AsyncErrorChannel, so bypassing it would leave an orphaned collector running instead of shutting down.

Fixed in 706ec46 with the if/else you suggested:

o.reportFunc = func(event *componentstatus.Event) {
    if _, ok := host.(componentstatus.Reporter); ok {
        componentstatus.ReportStatus(host, event)
        return
    }
    o.ComponentStatusChanged(selfInstanceID, event)
}

So the production path is preserved when the host implements componentstatus.Reporter, and the internal aggregator fallback only kicks in for non-Reporter hosts (where ReportStatus is a silent no-op). Added TestReportFuncPrefersHostReporter for the reporter path; the existing NopHost test still covers the fallback. Changelog subtext updated to match.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create an environment for this repo.

@CodeBlackwell

Copy link
Copy Markdown
Author

/easycla

Comment thread extension/opampextension/opamp_agent.go Outdated
// collector actually shuts down). Only when the host does not implement
// componentstatus.Reporter -- where ReportStatus silently no-ops -- fall
// back to this extension's own aggregator so the event is not dropped.
if _, ok := host.(componentstatus.Reporter); ok {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this branch does not preserve the real service shutdown path in production. Can you check?

I think we always fallback to o.ComponentStatusChanged

…anch

The service starts extensions with the bare graph.Host, which does not
implement componentstatus.Reporter (only pipeline components get a
Reporter-capable host wrapper), so the host.(Reporter) branch never ran in
production and its "preserves the service shutdown path" claim did not hold.
Always route self-reported events through the extension's own
ComponentStatusChanged handler so they reach the status aggregator, and drop
the now-obsolete reporter-preference test and misleading changelog subtext.
@CodeBlackwell

Copy link
Copy Markdown
Author

You're right, thanks for catching this. Traced it against collector core v0.156.0:

  • The service starts extensions with the bare graph.Host. Only pipeline components get the HostWrapper that implements componentstatus.Reporter.
  • So for an extension, host.(componentstatus.Reporter) is always false. We always hit the fallback, and the ReportStatus branch was dead code. The passing test only worked because it injected a synthetic Reporter host that production never supplies.
  • Pushed a commit that drops the dead branch and always routes through ComponentStatusChanged, so the event reliably reaches the aggregator and the orphan gets reported unhealthy. Removed the misleading test and changelog wording.

Remaining gap: this makes the orphan reported unhealthy but not actually shut down, and an extension has no direct line to the service's fatal-error channel, so real shutdown-on-orphan looks like a bigger design question than this fix. Want me to open a separate follow-up issue for that and keep this PR scoped to the dropped status event?

@douglascamata

Copy link
Copy Markdown
Member

@CodeBlackwell can you explain to us what's happening here more clearly, ideally with some examples and in your own words? What is the problem you observe? What is the proposed solution? This seems heavily AI generated, as seen by the co-author commit trailer and lack of use of our PR template and I'm struggling to understand it.

@CodeBlackwell

CodeBlackwell commented Jul 10, 2026

Copy link
Copy Markdown
Author

I'm Happy to Explain, apologies if my previous explanations were unclear. Yes I do use Ai to format and speed up responses but I understand the human to human aspect too.

In shortest words,

  • TThe README says the ppid setting's error emission policy is: when the watched parent process is no longer running, the extension emits a fatal error, causing the collector to exit. That is the stated guarantee.

  • The current behavior in the case illustrated in this reproduction script does not follow said policy.

Thats all this is tending to. If its a red herring, My genuine apologies - While It was brought to my attention via AI, I did my best due diligence to verify that this was a legit deviation from stated guidelines and a valid PR.

Below is a reproduction script

#!/bin/bash
# Repro: opamp ppid watchdog should make an orphaned collector exit. It doesn't.
set -u
cd "$(dirname "$0")"
VER=0.156.0

# Get the collector binary if needed
if ! ./otelcol-contrib --version 2>/dev/null; then
    OS=$(uname | tr '[:upper:]' '[:lower:]')
    ARCH=$(uname -m); [ "$ARCH" = "x86_64" ] && ARCH=amd64
    echo "Downloading otelcol-contrib v${VER} (${OS}_${ARCH})..."
    curl -sL "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${VER}/otelcol-contrib_${VER}_${OS}_${ARCH}.tar.gz" | tar xz otelcol-contrib
    ./otelcol-contrib --version || exit 1
fi

# Collector config: watch PARENT_PID, poll every 1s
cat > config.yaml <<'EOF'
extensions:
  opamp:
    server:
      ws:
        endpoint: wss://127.0.0.1:4320/v1/opamp
    ppid: ${env:PARENT_PID}
    ppid_poll_interval: 1s

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317

exporters:
  debug:

service:
  extensions: [opamp]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]
EOF

# Fake parent process
sleep 600 &
export PARENT_PID=$!
disown
echo "parent pid: $PARENT_PID"

# Start collector watching the parent
./otelcol-contrib --config config.yaml > collector.log 2>&1 &
COLLECTOR_PID=$!
sleep 3
ps -p $COLLECTOR_PID > /dev/null || { echo "collector died on startup, see collector.log"; exit 1; }
echo "collector pid: $COLLECTOR_PID"

# Orphan the collector. Docs say: fatal error + exit within ~1s.
echo "killing parent..."
kill $PARENT_PID
sleep 5

# Verdict
if ps -p $COLLECTOR_PID > /dev/null; then
    echo "BUG: collector still running 5s after being orphaned"
else
    echo "collector exited as documented (bug not reproduced)"
fi
if grep -qi orphan collector.log; then
    grep -i orphan collector.log
else
    echo "BUG: 'orphaned' fatal error never logged"
fi

kill $COLLECTOR_PID 2>/dev/null
image

@CodeBlackwell

Copy link
Copy Markdown
Author

Also I'm Having an issue with the easycla. I've signed it multiple times but it wont clear the error.

@douglascamata

Copy link
Copy Markdown
Member

@CodeBlackwell thanks for the explanation. I think I understand this better now. I also like the reproduction script, that's helpful.

I don't think your solution works though. It will push that status, yes. But it won't ultimately cause the Collector to exit when the PPID is gone. Can you check this?

@douglascamata

Copy link
Copy Markdown
Member

Yes I do use Ai to format and speed up responses but I understand the human to human aspect too.

@CodeBlackwell no problem using AI to tidy up text or to write the code. It's also fine to write "my AI said (...)" or add a small note saying "I used AI to format the text, but the contents itself are all written by" in the PR. It's just about disclosure. 👍

@douglascamata

Copy link
Copy Markdown
Member

Regarding EasyCLA, I don't know how to handle this. I'm not even sure how to debug it. We can try to get some help from maintainers that have more experience with it.

@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Jul 18, 2026

Copy link
Copy Markdown

Pull request dashboard status

Status last refreshed: 2026-07-27 15:23:04 UTC.

  • Waiting on: Author
  • Next step: Address or respond to 2 review feedback items:
    • Inline threads: 1
    • Top-level feedback: 2
    • For each item, reply to move the discussion forward, e.g. link to the commit that addresses it, explain why no change is needed, or ask a follow-up question.

This automated status or its linked feedback items may be incorrect. If something looks wrong, please report it with the result you expected. If you believe this pull request is incorrectly routed as waiting on the author, comment /dashboard route:reviewers to route it from waiting on the author to waiting on reviewers. If the last refreshed time above predates your latest reply or push, the dashboard hasn't processed it yet.

@CodeBlackwell
CodeBlackwell force-pushed the crg-farm/opamp-agent-fix branch from 47f6636 to ebb884c Compare July 22, 2026 00:33
@CodeBlackwell

CodeBlackwell commented Jul 22, 2026

Copy link
Copy Markdown
Author

You are correct, it doesn't exit. I dug through collector core v0.156.0 to check and the exit path just isn't reachable from an extension.

In shortest words:

  • The service starts extensions with the bare graph.Host, which doesn't implement componentstatus.Reporter - so ReportStatus from an extension is a silent no-op.
  • The machinery that actually shuts the collector down (fatal event -> AsyncErrorChannel) only hangs off HostWrapper (internal/graph/graph.go:445), and only pipeline components ever get that wrapper. Extensions never see it.

So making the collector truly exit needs a fix in core, not contrib. Best this PR can do is what it does now - the orphan shows up as unhealthy instead of the event being silently dropped.

I suppose A PR Name change?

I solved the CLA problem.

@douglascamata

Copy link
Copy Markdown
Member

So making the collector truly exit needs a fix in core, not contrib. Best this PR can do is what it does now - the orphan shows up as unhealthy instead of the event being silently dropped.

Sounds good, @CodeBlackwell. We can move forward with this fix and open an issue in core to fix the wrapping. Then when core is fixed this logic will behave as expected. 👍

I suppose A PR Name change?

Personally I think the PR name is fine now, but feel free to change it if you have better ones in mind.

@douglascamata

Copy link
Copy Markdown
Member

/workflow-approve

@douglascamata

Copy link
Copy Markdown
Member

@CodeBlackwell please keep in mind we have a pull request template to be followed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants