Weight-class configuration for LeastWeighted target allocation strategy

[suprjinx]

Description:

This PR implements weight-based target allocation for the least-weighted strategy, so that heavier targets are spread across collectors rather than piling up on one. In the absence of weight information, the strategy falls back to existing NumTargets balancing.

How to assign weights to targets

Add the opentelemetry.io/target-allocation-weight annotation to the target pod with an integer value from 1 to 100:

metadata:
  annotations:
    opentelemetry.io/target-allocation-weight: "10"

For pods you don't control (e.g. kube-state-metrics, nginx-ingress), you can patch the deployment's pod template:

kubectl patch deployment nginx-ingress -p '{"spec":{"template":{"metadata":{"annotations":{"opentelemetry.io/target-allocation-weight":"10"}}}}}'

Kubernetes SD automatically surfaces this as the meta label __meta_kubernetes_pod_annotation_opentelemetry_io_target_allocation_weight, which the allocator reads directly.

Weight resolution

1. Pod annotation value (integer 1-100) — if present and valid
2. Default: 1

Values outside the 1-100 range, non-numeric, or missing annotations all fall back to the default weight of 1.

How it works

The least-weighted strategy tracks WeightedLoad (sum of weights) per collector instead of just target count. When assigning a new target, it picks the collector with the lowest WeightedLoad. This ensures heavy targets get spread across collectors.

Link to tracking Issue(s):

• Relates: An algorithm for new target allocation, based on the average job algorithm. #3128

Testing:
Unit tests covering: weighted load balancing, unlabeled targets defaulting to weight 1, invalid annotations (non-numeric, zero, negative, >100), and balanced distribution of heavy targets.

Documentation:
Needed

PR Assisted by Claude Code

[suprjinx]

this is a significant change -- the filtered labels were not persisted to the item previously

[jaronoff97]

I understand the goal of this PR, but this change could indeed be decently dangerous. I have to spend some time thinking through the consequences of this. cc @swiatekm

[swiatekm]

There's no way we can do this, it's a major breaking change. My original thought was that we were going to use a Pod annotation, because we get those straight from K8s service discovery, and don't need relabeling.

[suprjinx]

There's no way we can do this, it's a major breaking change. My original thought was that we were going to use a Pod annotation, because we get those straight from K8s service discovery, and don't need relabeling.

Thanks @swiatekm @jaronoff97 -- I will remove the relabeling. I can use pod annotation, but i think it will also require some new config option for TA (in order to supply a weight config for Pods we don't control)

[suprjinx]

Ready for another look, with TA CR changes for WeightOverrides

[github-actions]

E2E Test Results

 34 files  ±0  247 suites  ±0   3h 0m 45s ⏱️ + 43m 44s
 95 tests ±0   94 ✅  - 1  0 💤 ±0  1 ❌ +1 
251 runs  ±0  250 ✅  - 1  0 💤 ±0  1 ❌ +1 

For more details on these failures, see this check.

Results for commit d13f23b. ± Comparison against base commit ebb6ab9.

♻️ This comment has been updated with latest results.

[swiatekm]

The changes in this PR look broadly correct, but do more than I'm comfortable with without further discussion about the API and how this feature should work in the long term. Originally, the problem we wanted solved was that jobs of a particular class should be evenly spread amongst collectors. The simple solution I envisioned for this was to set the class as a Pod annotation or label and let the strategy use that information, the same way it currently does with the job name.

Here, we're now adding CR-level overrides and tracking weight as if it's a meaningful number. I'm not completely against doing that, but it requires a more careful design, especially if we're going to make it configurable at the CR level.

@suprjinx do you think we can cut this down to a more basic version while still keeping it useful for you?

[suprjinx]

The changes in this PR look broadly correct, but do more than I'm comfortable with without further discussion about the API and how this feature should work in the long term. Originally, the problem we wanted solved was that jobs of a particular class should be evenly spread amongst collectors. The simple solution I envisioned for this was to set the class as a Pod annotation or label and let the strategy use that information, the same way it currently does with the job name.

Here, we're now adding CR-level overrides and tracking weight as if it's a meaningful number. I'm not completely against doing that, but it requires a more careful design, especially if we're going to make it configurable at the CR level.

@suprjinx do you think we can cut this down to a more basic version while still keeping it useful for you?

@swiatekm so you'd suggest consuming an annotation, configured in podSpec, but no ability to configure weight for kube-state-metrics etc? I think the latter is where we see the greatest imbalance in load on collectors.

[swiatekm]

The changes in this PR look broadly correct, but do more than I'm comfortable with without further discussion about the API and how this feature should work in the long term. Originally, the problem we wanted solved was that jobs of a particular class should be evenly spread amongst collectors. The simple solution I envisioned for this was to set the class as a Pod annotation or label and let the strategy use that information, the same way it currently does with the job name.
Here, we're now adding CR-level overrides and tracking weight as if it's a meaningful number. I'm not completely against doing that, but it requires a more careful design, especially if we're going to make it configurable at the CR level.
@suprjinx do you think we can cut this down to a more basic version while still keeping it useful for you?

@swiatekm so you'd suggest consuming an annotation, configured in podSpec, but no ability to configure weight for kube-state-metrics etc? I think the latter is where we see the greatest imbalance in load on collectors.

If what you're actually trying to do is balance based on a statically assigned numeric load, so whichever collector gets kube-state-metrics doesn't get a lot of other targets, then I'm not sure we want to do that. It's a fairly specific use case and something of an anti-pattern for Prometheus in general - you should try to avoid having massive targets. I don't think an unreasonable suggestion would be to either run a separate collector instance just for kube-state-metrics or shard it, either as a StatefulSet or Daemonset. Is that not an option for you?

[suprjinx]

The changes in this PR look broadly correct, but do more than I'm comfortable with without further discussion about the API and how this feature should work in the long term. Originally, the problem we wanted solved was that jobs of a particular class should be evenly spread amongst collectors. The simple solution I envisioned for this was to set the class as a Pod annotation or label and let the strategy use that information, the same way it currently does with the job name.
Here, we're now adding CR-level overrides and tracking weight as if it's a meaningful number. I'm not completely against doing that, but it requires a more careful design, especially if we're going to make it configurable at the CR level.
@suprjinx do you think we can cut this down to a more basic version while still keeping it useful for you?

@swiatekm so you'd suggest consuming an annotation, configured in podSpec, but no ability to configure weight for kube-state-metrics etc? I think the latter is where we see the greatest imbalance in load on collectors.

If what you're actually trying to do is balance based on a statically assigned numeric load, so whichever collector gets kube-state-metrics doesn't get a lot of other targets, then I'm not sure we want to do that. It's a fairly specific use case and something of an anti-pattern for Prometheus in general - you should try to avoid having massive targets. I don't think an unreasonable suggestion would be to either run a separate collector instance just for kube-state-metrics or shard it, either as a StatefulSet or Daemonset. Is that not an option for you?

@swiatekm I removed all the API changes so this is now just an optional annotation. Also, now expecting a numeric weight (1-100) rather than a class name.