Skip to content

Commit 4cc505e

Browse files
bhperrybhperry
authored
Proposal for placement sort order and score visibility (#143)
* Proposal for placement sort order and score visibility Signed-off-by: Ben Perry <bhperry94@gmail.com> * Update metadata Signed-off-by: Ben Perry <bhperry94@gmail.com> * Add score rate limit Signed-off-by: Ben Perry <bhperry94@gmail.com> * ScoreRateLimit validation and default Signed-off-by: Ben Perry <bhperry94@gmail.com> --------- Signed-off-by: Ben Perry <bhperry94@gmail.com>
1 parent 5b5f16a commit 4cc505e

2 files changed

Lines changed: 214 additions & 0 deletions

File tree

‎enhancements/sig-architecture/225-placement-score-visibility/README.md‎

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
# Placement Score Visibility
2+
3+
## Release Signoff Checklist
4+
5+
- [] Enhancement is `provisional`
6+
- [] Design details are appropriately documented from clear requirements
7+
- [] Test plan is defined
8+
- [] Graduation criteria for dev preview, tech preview, GA
9+
- [] User-facing documentation is created in [website](https://github.com/open-cluster-management-io/open-cluster-management-io.github.io/)
10+
11+
## Summary
12+
13+
This proposal extends existing Placement mechanisms to surface the scores of each cluster
14+
in PlacementDecision so that users (or future OCM features) can make advanced scheduling decisions
15+
based on the scored state of clusters.
16+
17+
In addition, it adds support for different sort strategies on the decisions generated by a Placement.
18+
By default, decisions will continue to be sorted by "ClusterName". An additional sort strategy "Score" will
19+
enable sorting decisions numerically in descending order by prioritizer score.
20+
21+
To avoid creating too much API pressure from scores changing, the field ScoreRateLimit will restrict
22+
the frequency of changes to placements.
23+
24+
## Motivation
25+
26+
Placement scores are a powerful feature of OCM for scheduling workloads, but there is currently
27+
no reliable way for users to see what scores were assigned to clusters for a placement. Scores are
28+
reported in events, but these are ephemeral. Since PlacementDecisions are orderd alphanumerically
29+
there is no good way to see the relative scores of clusters in a decision other than recomputing
30+
them for each cluster in the decision group(s).
31+
32+
A user may wish to use Placement to find all clusters that match some conditions, but then pick
33+
only the top N clusters for their workload. While this could be accomplished by setting
34+
`numberOfClusters: N` on the placement, this means a new placement must be created for every value
35+
of N. Another use-case would be to distribute workloads out to selected clusters randomly weighted
36+
by their scores. There is no way to do this without recomputing scores for each cluster
37+
listed in the PlacementDecisions. Displaying scores in an easy to consume manner gives more flexibility
38+
to the user, and enables a single Placement to be reused for many different scheduling decisions
39+
(reducing API pressure on the hub).
40+
41+
## Proposal
42+
43+
Add `SortBy` and `ScoreRateLimit` to `PlacementSpec` to configure the sort order and rate limit of
44+
score updates for PlacementDecision. This field is an enum with values "ClusterName" and "Score".
45+
Add `LastScoreUpdateTime` to `PlacementStatus` to track the last time scores were updated for the placement.
46+
Add `Score` to `ClusterDecision` to reflect the latest score of each cluster in the decision.
47+
48+
### Design Details
49+
50+
#### API change
51+
52+
Fields for sorting decisions and rate limiting score updates
53+
54+
```go
55+
type PlacementSpec struct {
56+
...
57+
58+
// SortBy sets the sort order for decisions.
59+
// It can be "ClusterName", "Score", or "", where "" is "ClusterName" by default.
60+
// If sortBy is "ClusterName", decisions will be orderd alphanumerically by cluster name
61+
// If sortBy is "Score", decisions will be ordered numerically in descending order by score,
62+
// then by cluster name in the event of a tie
63+
// +kubebuilder:default:=ClusterName
64+
// +optional
65+
SortBy PlacementSortByType `json:"sortBy,omitempty"`
66+
67+
// ScoreRateLimit sets maximum rate of updates to recorded scores in placement decisions.
68+
// Score changes that do not change the selected set of clusters in the placement will
69+
// not be reflected in decisions more often than the given duration.
70+
// +kubebuilder:validation:Pattern="^[0-9]+[h|m|s]$"
71+
// +kubebuilder:default:="1m"
72+
// +optional
73+
ScoreRateLimit string `json:"scoreRateLimit,omitempty"`
74+
}
75+
76+
// +kubebuilder:validation:Enum=ClusterName;Score
77+
type PlacementSortByType string
78+
79+
const (
80+
// PlacementSortByClusterName sorts decisions alphanumerically by cluster name
81+
PlacementSortByClusterName PlacementSortByType = "ClusterName"
82+
// PlacementSortByScore sorts decisions numerically in descending order by score.
83+
// If one or more clusters are tied on score they will be sorted by name
84+
PlacementSortByScore PlacementSortByType = "Score"
85+
)
86+
87+
type PlacementStatus struct {
88+
...
89+
90+
// LastScoreUpdateTime records the last time that placement scores were updated.
91+
LastScoreUpdateTime *metav1.Time `json:"lastScoreUpdateTime"`
92+
}
93+
```
94+
95+
Display score to consumer of decisions
96+
97+
```go
98+
type ClusterDecision struct {
99+
...
100+
101+
// Score is the computed score for the cluster based on configured prioritizers
102+
Score int64 `json:"score"`
103+
}
104+
105+
```
106+
107+
#### hub implementation
108+
109+
In `scheduling_controller.go`, update `divideDecisionGroups` to check the value of `SortBy` before
110+
sorting and splitting clusters into groups. Any pre-determined decision groups set in
111+
`placement.Spec.DecisionStrategy.GroupStrategy.DecisionGroups` will be individually ordered within their
112+
named groups by the `SortBy` strategy. Any remaining clusters will be sorted across all remaining
113+
groups before being split.
114+
115+
Prioritizer scores will be passed along with the clusters slice so that each cluster's score can
116+
be looked up when generating `clusterapiv1beta1.ClusterDecision`.
117+
118+
Update `schedulingController.bind` and `schedulingController.createOrUpdatePlacementDecision`
119+
to check what type of changes exist between old and new cluster decisions. If only scores have changed
120+
then the updates are applied based on the configured rate limit compared to lastScoreUpdateTime
121+
on the placement. If there are any other changes to the decisions (e.g. added or removed clusters),
122+
then the update is triggered regardless of rate limit.
123+
124+
#### examples
125+
126+
Sort placement decisions by score with rate limit
127+
128+
```yaml
129+
apiVersion: cluster.open-cluster-management.io/v1beta1
130+
kind: Placement
131+
metadata:
132+
name: h100
133+
namespace: example
134+
spec:
135+
clusterSets:
136+
- global
137+
predicates:
138+
- requiredClusterSelector:
139+
labelSelector:
140+
matchLabels:
141+
gpu.example.com/h100: enabled
142+
prioritizerPolicy:
143+
configurations:
144+
- scoreCoordinate:
145+
addOn:
146+
resourceName: hardware-availability
147+
scoreName: h100
148+
type: AddOn
149+
weight: 1
150+
sortBy: Score
151+
scoreRateLimit: 1m
152+
---
153+
apiVersion: cluster.open-cluster-management.io/v1beta1
154+
kind: PlacementDecision
155+
metadata:
156+
labels:
157+
cluster.open-cluster-management.io/decision-group-index: "0"
158+
cluster.open-cluster-management.io/decision-group-name: ""
159+
cluster.open-cluster-management.io/placement: h100
160+
name: h100-decision-1
161+
namespace: example
162+
ownerReferences:
163+
- apiVersion: cluster.open-cluster-management.io/v1beta1
164+
controller: true
165+
kind: Placement
166+
name: h100
167+
status:
168+
decisions:
169+
- clusterName: cluster-2
170+
score: 80
171+
- clusterName: cluster-1
172+
score: 65
173+
- clusterName: cluster-5
174+
score: 10
175+
- clusterName: cluster-6
176+
score: 10
177+
```
178+
179+
180+
### Test Plan
181+
- test that the default alphanumeric sort order remains the same
182+
- test that `sortBy: Score` properly orders clusters across decision groups
183+
- test that named decision groups are sorted individually based on `sortBy`
184+
- test that decision groups are updated when scores change
185+
- test that score-only updates are not applied more often than the rate limit allows
186+
187+
### Graduation Criteria
188+
N/A
189+
190+
### Upgrade Strategy
191+
It will need upgrade on CRD of Placement and PlacementDecision on hub cluster. No changes to agents.
192+
193+
When a user needs to use this feature with an existing `Placement`, the user needs to add
194+
`sortBy: Score` to their placements. No changes are required to update existing decisions
195+
with score (regardless of sortBy). Once the placement controller is updated, all
196+
`PlacementDecisions` will be updated with prioritizer scores on their next reconciliation.
197+
198+
### Version Skew Strategy
199+
- The new fields are optional, and if not set, the placement decisions will be ordered the same
200+
as in previous versions.
201+
- Older versions of the placement controller will ignore the newly added fields

‎enhancements/sig-architecture/225-placement-score-visibility/metadata.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
title: placement-score-visibility
2+
authors:
3+
- "@bhperry"
4+
reviewers:
5+
- "@deads2k"
6+
- "@elgnay"
7+
- "@zhujian7"
8+
approvers:
9+
- "@elgnay"
10+
creation-date: 2025-05-22
11+
last-updated: 2025-05-22
12+
status: provisional
13+
see-also: []

0 commit comments

Comments
 (0)