feat: add Prometheus metrics for operator reconciliation

[Chaithanya5gif]

Description

This PR fixes #779

Add Prometheus metrics to Meshery Operator for observability:

• meshery_operator_reconcile_total — total reconciliations by controller
• meshery_operator_reconcile_errors_total — total errors by controller
• meshery_operator_reconcile_duration_seconds — reconciliation duration by controller

Changes:

• New pkg/metrics package with metric definitions
• Instrument BrokerReconciler with timing and error tracking
• Instrument MeshSyncReconciler with timing and error tracking
• Tests for metric registration

Notes for Reviewers:

• Metrics follow controller-runtime patterns
• Uses metrics.Registry.MustRegister for automatic exposition
• Both controllers instrumented with defer pattern for consistent tracking

Signed commits

• Yes, I signed my commits.

[welcome]

Yay, your first pull request! 👍 A contributor will be by to give feedback soon. In the meantime, you can find updates in the #github-notifications channel in the community Slack.
Be sure to double-check that you have signed your commits. Here are instructions for making signing an implicit activity while performing a commit.

[leecalcote]

Thanks for this, the reconcile counters and duration histogram are exactly the kind of observability we want on the operator.

We are keeping this and folding it into Workstream 5 (metrics and TLS hardening), which adds authn/authz-protected metrics serving on the modernized controllers. Heads up that Workstreams 2 and 3 are reworking BrokerReconciler and MeshSyncReconciler under the new Kubebuilder go/v4 layout, so we will ask you to rebase the instrumentation onto the updated controller signatures once those land. The defer-based timing pattern and the metric naming look good as-is.

We will keep this open and iterate rather than supersede it.

[leecalcote]

Things are looking pretty good here, @Chaithanya5gif. 👍

A few merge conflicts have popped up, but outside of those I think we're about ready to merge. We need some documentation in Meshery Docs on the fact that the Operator is now emitting these metrics for monitoring. Please add to the Production Deployment docs and hyperlink your issue and PR on the docs here, please.

[leecalcote]

Thanks again, @Chaithanya5gif. I reviewed the changes in detail and pulled the reconciliation instrumentation forward onto the current master (post Kubebuilder go/v4, WS-1) in #793, crediting you — the metric naming and the defer-based timing pattern carry over unchanged.

A few notes from the review:

• The Makefile and CONTRIBUTING.md edits are no longer needed: WS-1 added an envtest-aware make test target and docs/testing.md, which already cover the etcd not found in PATH guidance you documented, so feat: add Prometheus metrics for operator reconciliation #793 leaves those files untouched.
• metrics_test.go's import grouping would trip the gci/goimports formatters in CI, and the repeated "controller"/"broker"/"meshsync" label literals would trip goconst. feat: add Prometheus metrics for operator reconciliation #793 pulls those into named constants and adds the Apache license headers the repo requires on new Go files.
• I switched the two reconcilers to named return values so the timing/error defer sits once at the top of Reconcile instead of rewriting every return site, and extended the tests to assert the metrics are actually registered with the controller-runtime registry (the acceptance criterion).

This keeps us moving toward folding the work into Workstream 5 (metrics + TLS hardening). Once WS-2/WS-3 reshape the reconciler signatures, we'll rebase once more on top of those. Appreciate you kicking this off. 🙏

[github-actions]

Checking in... it has been awhile since we've heard from you on this issue. Are you still working on it? Please let us know and please don't hesitate to contact a MeshMate or any other community member for assistance.

---

        Be sure to join the community, if you haven't yet and please leave a ⭐ star on the project 😄

[Chaithanya5gif]

@leecalcote Thanks for the detailed review and for pulling this forward in #793.

I see the merge conflicts since you've already integrated the metrics work into #793, should I close this PR and focus on the documentation follow up you requested?

I'll work on adding the Operator metrics monitoring guide to the Production Deployment docs and open a separate docs PR.