[target-allocator] Add feature flag for labeled targets_remaining metric with OTel conventions

[hkf57]

Description

Adds k8s.namespace.name and job.name labels to the opentelemetry_allocator_targets_remaining metric (when enabled via feature gate) to enable distinguishing between intentionally filtered targets and orphaned targets due to bugs.

Breaking Change Mitigation: This change is behind the operator.targetallocator.labeledmetrics feature gate (alpha) to provide a migration path for existing users.

Motivation and Context

Currently, opentelemetry_allocator_targets_remaining reports a scalar count, making it impossible to create meaningful alerts. In production environments with namespace filtering (e.g., excluding istio-system to control costs), the baseline can be ~1600 targets. This masks genuine issues like orphaned targets from deleted collectors or version skew scenarios.

Problem Example

In production environments:

• Filtering istio-system namespace (~1600 targets) via allowNamespaces to reduce costs
• Recent 14-hour outage where kube-state-metrics target was orphaned due to version skew during StatefulSet rollout
• Could not alert on orphaned targets because targets_remaining baseline was already 1600

Solution

Feature Flag (Alpha)

Added operator.targetallocator.labeledmetrics feature gate:

• Default: Disabled (maintains backward compatibility)
• When enabled: Emits metrics with OTel semantic convention labels
• Stage: Alpha
• From version: v0.145.0

Label Names (OTel Semantic Conventions)

Following OpenTelemetry semantic conventions:

• job.name (was job_name)
• k8s.namespace.name (was namespace)

Feature Gate Propagation

Follows the existing operator pattern for feature gate propagation to the target allocator:

• Operator checks operator.targetallocator.labeledmetrics gate
• Writes labeled_metrics: true into the TA configmap when enabled
• TA reads from its config file (same pattern as allocation_fallback_strategy and https)
• No CLI flag forwarding needed; config-based propagation is the established convention

Implementation

Modified SetTargets() in allocator.go to:

1. Check labeled_metrics config value
2. When enabled: Group remaining targets by job and namespace, emit labelled metrics
3. When disabled: Emit single unlabelled gauge (original behaviour)
4. Extract namespace from __meta_kubernetes_namespace label
5. Use "unknown" for targets without namespace metadata

Usage

Enable via the operator's feature gates:

# On the operator
--feature-gates=operator.targetallocator.labeledmetrics

The operator will propagate this to the target allocator's configmap automatically.

After this change (when feature enabled)

Alerts can filter specific namespaces:

# Alert on orphaned targets, excluding known filtered namespaces
opentelemetry_allocator_targets_remaining{k8s_namespace_name!="istio-system"} > 10

# Or alert on specific critical jobs
opentelemetry_allocator_targets_remaining{job_name="kube-state-metrics"} > 0

Testing

• Code compiles successfully
• Mock interfaces updated
• Feature gate registered in global registry
• Configmap propagation test
• Collector not ready grace period test updated for new config field
• Unit tests for labelled/unlabelled behaviour
• E2E tests with feature flag enabled/disabled

Checklist

• DCO sign-off included
• No breaking changes (behind feature flag)
• Follows OTel semantic conventions
• Follows existing feature gate propagation pattern (configmap, not CLI flags)
• Unit tests added
• E2E tests updated

Fixes #4637

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: hkf57 / name: Charles Lee (5d9c36b, 8947c65, dd71375, f7c97ab)

[swiatekm]

I don't think what you've done is enough for the feature gate to propagate down. Right now, if you set it on the operator, it won't do anything. It is possible to set it on the allocator directly by passing it as an argument, but that's quite inconvenient. It'd be better if the operator did it if it's enabled.