Skip to content

Commit b4f7289

Browse files
authored
fix(hubble): surface controller-runtime manager start failures (#2399)
# Description Wrap the controller-runtime manager's start call so that any returned error or an unexpected clean return is logged and triggers a hive shutdown instead of being silently discarded by the worker pool. ## Related Issue If this pull request is related to any issue, please mention it here. Additionally, make sure that the issue is assigned to you before submitting this pull request. ## Checklist - [x] I have read the [contributing documentation](https://retina.sh/docs/Contributing/overview). - [x] I signed and signed-off the commits (`git commit -S -s ...`). See [this documentation](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification) on signing commits. - [x] I have correctly attributed the author(s) of the code. - [x] I have tested the changes locally. - [ ] I have followed the project's style guidelines. - [ ] I have updated the documentation, if necessary. - [ ] I have added tests, if applicable. ## Screenshots (if applicable) or Testing Completed In my case, a port conflict was preventing the controller-runtime manager from starting: ``` ts=2026-06-02T13:35:23.949Z level=error caller=hubble/daemon_linux.go:86 msg="controller-runtime manager exited with error; node reconciler is no longer running" module=agent.control-plane.daemon error="failed to start metrics server: failed to create listener: listen tcp :18080: bind: address already in use" stacktrace="log/slog.(*Logger).Error\n\t/usr/local/go/src/log/slog/logger.go:229\ngithub.qkg1.top/microsoft/retina/cmd/hubble.init.func4.1.1\n\t/go/src/github.com/microsoft/retina/cmd/hubble/daemon_linux.go:86\ngithub.qkg1.top/cilium/workerpool.(*WorkerPool).run.func1\n\t/go/pkg/mod/github.com/cilium/workerpool@v1.4.0/workerpool.go:269" ``` ## Additional Notes Add any additional notes or context about the pull request here. --- Please refer to the [CONTRIBUTING.md](../CONTRIBUTING.md) file for more information on how to contribute to this project. Signed-off-by: Sébastien Larivière <sebastien.lariviere@goto.com>
1 parent b165b60 commit b4f7289

1 file changed

Lines changed: 18 additions & 2 deletions

File tree

cmd/hubble/daemon_linux.go

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ import (
2525
"github.com/cilium/cilium/pkg/k8s/watchers"
2626
monitoragent "github.com/cilium/cilium/pkg/monitor/agent"
2727
"github.com/cilium/cilium/pkg/node"
28+
"github.com/cilium/hive"
2829
"github.com/cilium/hive/cell"
2930
"github.com/cilium/workerpool"
3031

@@ -72,14 +73,29 @@ var (
7273
}),
7374

7475
// Start the controller manager
75-
cell.Invoke(func(l *slog.Logger, lifecycle cell.Lifecycle, ctrlManager ctrl.Manager) {
76+
cell.Invoke(func(l *slog.Logger, lifecycle cell.Lifecycle, shutdowner hive.Shutdowner, ctrlManager ctrl.Manager) {
7677
var wp *workerpool.WorkerPool
7778
lifecycle.Append(
7879
cell.Hook{
7980
OnStart: func(cell.HookContext) error {
8081
wp = workerpool.New(1)
8182
l.Info("starting controller-runtime manager")
82-
if err := wp.Submit("controller-runtime manager", ctrlManager.Start); err != nil {
83+
if err := wp.Submit("controller-runtime manager", func(ctx context.Context) error {
84+
// Shut down immediately if the manager exits unexpectedly to avoid running without reconciliation.
85+
if err := ctrlManager.Start(ctx); err != nil {
86+
l.Error("controller-runtime manager exited with error; node reconciler is no longer running", "error", err)
87+
shutdowner.Shutdown(hive.ShutdownWithError(err))
88+
return errors.Wrap(err, "running controller-runtime manager")
89+
}
90+
if ctx.Err() == nil {
91+
// Start returned early while the agent context is still active.
92+
err := errors.New("controller-runtime manager stopped unexpectedly")
93+
l.Error("controller-runtime manager stopped unexpectedly; node reconciler is no longer running")
94+
shutdowner.Shutdown(hive.ShutdownWithError(err))
95+
return err
96+
}
97+
return nil
98+
}); err != nil {
8399
return errors.Wrap(err, "failed to submit controller-runtime manager to workerpool")
84100
}
85101
return nil

0 commit comments

Comments
 (0)