Skip to content

Commit a14f0f3

Browse files
youngbuparkyoungbupark
authored
Initial PR for rollout plugin (#160)
Signed-off-by: Young Bu Park <youngbu.park@salesforce.com>
1 parent 2ece292 commit a14f0f3

2 files changed

Lines changed: 398 additions & 0 deletions

File tree

‎enhancements/sig-architecture/231-manifestworkreplicaset-rollout-plugin/README.md‎

Lines changed: 383 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,383 @@
1+
# ManifestWorkReplicaSet Rollout Plugin
2+
3+
## Release Signoff Checklist
4+
5+
- [x] Enhancement is `implementable`
6+
- [ ] Design details are appropriately documented from clear requirements
7+
- [ ] Test plan is defined
8+
- [ ] Graduation criteria for dev preview, tech preview, GA
9+
- [ ] User-facing documentation is created in [website](https://github.com/open-cluster-management-io/open-cluster-management-io.github.io/)
10+
11+
## Summary
12+
13+
The ManifestWorkReplicaSet (MWRS) Rollout plugin introduces a plugin-based extensibility model for the OCM Work Controller, enabling users to inject custom logic at critical phases of the multi-cluster rollout and rollback lifecycle.
14+
15+
This design allows user to implement domain-specific logic -- such as progressive traffic management, post-deployment validation, and automatic rollback -- without changing the core MWRS controller.
16+
17+
The rollout plugin framework provides a flexible and secure mechanism to orchestrate multi-cluster rollouts that integrate seamlessly with external components and services such as Argo Rollout, Istio / service meshes, or custom validation frameworks.
18+
19+
## Motivation
20+
21+
The ManifestWorkReplicaSet (MWRS) simplifies multi-cluster workload distribution by supporting several rollout strategies that enable gradual deployment of workloads across clusters.
22+
23+
In real-world use cases, rollout orchestration often requires cross-cluster coordination and dynamic control at each rollout phase. Examples include:
24+
25+
* Multi-cluster aware Argo Rollouts, where progressive canary traffic shifting must align with MWRS orchestration status.
26+
* Multi-cluster mesh traffic control, where traffic weights must be adjusted across clusters as rollout progresses.
27+
28+
To ensure safe rollouts, users need the ability to verify functionality at each cluster stage before promotion to the next cluster. This often involves executing custom post-deployment validation tests to prevent propagating a faulty deployment across clusters — beyond what can be expressed through existing `conditionRules`.
29+
30+
Finally, users also require automated rollback capabilities to revert to a previous revision when failures are detected. Rollback procedures often include cluster-specific cleanup or manifest mutation steps that are hard to standardize.
31+
32+
Because these rollout, validation, and rollback workflows are domain-specific, it is difficult to generalize them within the work controller itself. A flexible plugin mechanism is therefore needed to allow users to define custom logic for these operations.
33+
34+
### Goals
35+
36+
- Design a plugin architecture for the work-controller that supports custom hooks at various rollout phases.
37+
- Define the gRPC API contract and protocol for communication between the work-controller.
38+
- Define the configuration required to enable and load a custom plugin.
39+
40+
### Non-Goals
41+
42+
- Implementing any specific plugin (e.g., an Argo Rollouts or Istio plugin). This design focuses only on the plugin architecture and contract.
43+
- Designing the core rollback logic within the work-controller itself. The design will provide Rollback hooks, but the plugin is responsible for implementing the actual rollback orchestration logic.
44+
- Designing a manifest template revisioning system or history for the ManifestWorkReplicaSet.
45+
46+
## Proposal
47+
48+
### User Stories
49+
50+
#### Story 1 - multi-cluster aware argo rollout
51+
52+
As a user, I want to perform progressive canary rollouts across multiple clusters using existing Argo Rollout resources.
53+
During the rollout, I need each cluster to understand its position in the global rollout sequence (for example, cluster 3 of 10) and adjust its behavior — such as traffic weights — accordingly.
54+
This coordination should happen automatically based on MWRS orchestration progress, so that rollout across clusters can be synchronized and traffic shifts can occur safely.
55+
56+
#### Story 2 - safe multi-cluster rollout
57+
58+
As a user, I want to pause or stop rollout immediately if a newly deployed version fails validation in one of the clusters.
59+
After each cluster completes its deployment, I need to run post-deployment tests (such as integration or smoke tests) before continuing to the next cluster. If a test fails, MWRS should stop further rollouts to prevent cascading impact.
60+
61+
#### Story 3 — automated rollback on failure
62+
63+
As a user, I want MWRS to automatically roll back workloads in the event of a failed rollout. When a deployment fails, MWRS should identify the previous revision and restore it consistently to the already rolled out clusters. The rollback should be able to perform any additional operations needed to safely revert to the stable version - for example, reverting manifests or skipping rollout steps in dependent systems in Argo Rollout use-case, which requires the mutation of old manifest resources.
64+
65+
## Design Details
66+
67+
```mermaid
68+
flowchart LR
69+
%% --- Hub Cluster Section ---
70+
subgraph Hub[Hub Cluster]
71+
direction TB
72+
subgraph WorkController[Work-controller Pod]
73+
MWRSController[work-controller]
74+
Plugin[Rollout Plugin]
75+
MWRSController -- gRPC --> Plugin
76+
end
77+
78+
subgraph NamespaceDefault[Namespace: default]
79+
MWRSController -->HelloRollout[MWRS: HelloRollout]
80+
MWRSController -->HelloPlacement[Placement: HelloPlacement]
81+
end
82+
83+
subgraph NamespaceCluster1[Namespace: cluster1]
84+
MWRSController -->|Apply/Delete| MW1[ManifestWork: HelloRollout]
85+
end
86+
87+
subgraph NamespaceCluster2[Namespace: cluster2]
88+
MWRSController -->|Apply/Delete| MW2[ManifestWork: HelloRollout]
89+
end
90+
end
91+
92+
MeshService[Istio ControlPlane]
93+
94+
Plugin -->|Update traffic weights| MeshService
95+
```
96+
97+
The design extends the work-controller with the plugin, which are implemented using a gRPC protocol between the controller and a plugin sidecar.
98+
99+
### Plugin protocol
100+
101+
The Plugin will be implemented as a sidecar container running alongside the work controller in hub cluster. We chose a sidecar model to simplify the plugin onboarding process. gRPC will be used for communication. The plugin will run as a gRPC server, and the work controller will act as the gRPC client.
102+
103+
#### Plugin initialization
104+
105+
When the work controller at hub starts, it attempts to call the `Initialize()` gRPC endpoint on the plugin server.
106+
107+
#### Rollout sequence with plugin
108+
109+
The following sequence diagram describes the high-level orchestration with Plugin API calls.
110+
111+
```mermaid
112+
sequenceDiagram
113+
participant APIServer
114+
participant Work
115+
participant PluginServer
116+
117+
loop Placements
118+
Work->>APIServer: Get all Manifestworks associated with the current placement
119+
Work->>Work: Get the desire revision (e.g. use .status.currentRevision for abort operation)
120+
loop existing manifests
121+
Work->>Work: Check if this manifest is same as the desired revision?
122+
Work->>Work: Determine cluster rollout status
123+
alt RolloutStatus is "Validating"
124+
Work->>PluginServer: (NEW) ValidateRolloutCompletion()
125+
PluginServer-->>Work: Status: SUCCEEDED/FAILED/INPROGRESS
126+
Work->>Work: Set RolloutStatus to SUCCEEDED if Validation result is SUCCEEDED
127+
end
128+
end
129+
Work->>Work: Create rollout handler
130+
Work->>Work: Find rollout/removed/timeout candidate clusters (RolloutResult)
131+
alt timeout clusters exists and .spec.placementRefs[*].rolloutStrategy.abortOnFailure is true
132+
Note over Work, PluginServer: Start automatic abort
133+
Work->>Work: Set .status.abort to true
134+
Work->>Work: Set .status.abortedTime to the current time
135+
Work->>Work: Update the desired revision to `.status.currentRevision`
136+
end
137+
Work->>PluginServer: (NEW) ProgressRollout()
138+
PluginServer-->>Work: OK
139+
loop clusterToRollout clusters
140+
Work->>PluginServer: (NEW) BeginRollout()
141+
PluginServer-->>Work: OK
142+
Work->>PluginServer: (NEW) MutateManifestwork(the desired revision)
143+
PluginServer-->>Work: (NEW) Return mutated Manifestwork resource
144+
Work->>APIServer: Apply the mutated ManifestWork resource to the current cluster
145+
end
146+
Work->>APIServer: Clean up manifestworks for removed clusters.
147+
end
148+
149+
```
150+
151+
This workflow introduces four new plugin API calls:
152+
153+
* `BeginRollout()`: Called before applying the ManifestWork to a target cluste to roll out new revision, allowing the plugin to perform any necessary preparations.
154+
* `ProgressRollout()`: Called during every reconciliation loop to report the current rollout status to the plugin.
155+
* `ValidateRolloutCompletion()`: Called after the `Progressing` condition on the target cluster's ManifestWork becomes `False` and `clsRolloutStatus.LastTransitionTime` will be set. This hook enables post-rollout testing before the status is set to Succeeded.
156+
- The work reconciler polls this hook periodicaly until the current time reaches to `clsRolloutStatus.LastTransitionTime+progressingDeadline`.
157+
- For example, the plugin could use this call to create a new ManifestWork that runs a Kubernetes Job for rollout validation.
158+
* `MutateManifestWork()`: Called before applying the `ManifestWork` to the cluster. This hook allows the plugin to modify the manifest, which is essential for:
159+
- Injecting orchestration status (like the cluster's rollout index) into resource labels or annotations.
160+
- Enabling advanced scenarios, such as modifying an Argo Rollout resource to skip steps during a rollback.
161+
- This direct mutation is required because it is difficult to inject this context via a default admission webhook without exposing all orchestration information in the ManifestWorkReplicaSet status.
162+
163+
The overall Rollout Status is determined by the ManifestWork conditions and the validation result:
164+
165+
| Progressing (ConditionRule) | Degraded (ConditionRule) | ValidateRolloutCompletion() | Rollout Status | Description |
166+
|---|---|---|---|---|
167+
| True | True | N/A | Failed | Work is progressing but degraded |
168+
| True | False or not set | N/A | Progressing | Work is being applied and is healthy |
169+
| False | False or not set | INPROGRESS | Validating | Validating rollout |
170+
| False | False or not set | FAILED | Failed | The current rollout is failed |
171+
| False | False or not set | SUCCEEDED | Succeeded | Work has been successfully applied |
172+
| Unknown/Not set | Any | N/A | Progressing | Conservative fallback: treat as still progressing |
173+
174+
The following state machine shows the expected transitions between rollout statuses:
175+
176+
```mermaid
177+
stateDiagram
178+
Validating: (NEW State) Validating
179+
NextCluster: Move to the next cluster rollout
180+
Abort: Abort (stop the current rollout and revert it back)
181+
[*] --> ToApply
182+
ToApply --> Progressing: Progressing = True
183+
Progressing --> Validating: Progressing = False
184+
Validating --> Succeeded: ValidateRolloutCompletion() returns SUCCEEDED
185+
Validating --> Failed: ValidateRolloutCompletion() returns FAILED
186+
Progressing --> Failed: Degraded = True
187+
ToApply --> Failed: Degraded = True
188+
Succeeded --> NextCluster
189+
Failed --> Abort
190+
```
191+
192+
### gRPC API design
193+
194+
The following service defines the contract between Work Controller and the plugin. Each call must be idempotent, stateless, and time-bounded (≤30 s) to ensure consistent controller reconciliation. Plugin server must implement the following APIs. The helpers to implement server and clients will be implemented in [ocm/sdk-go](https://github.com/open-cluster-management-io/sdk-go) repository.
195+
196+
```proto
197+
// RolloutPluginService is the service for the rollout plugin.
198+
service RolloutPluginService {
199+
// Initialize initializes the plugin.
200+
rpc Initialize(InitializeRequest) returns (InitializeResponse);
201+
202+
// BeginRollout is called before the manifestwork resource is applied.
203+
// It is used to prepare the rollout.
204+
rpc BeginRollout(RolloutPluginRequest) returns (google.protobuf.Empty);
205+
206+
// ProgressRollout is called after the manifestwork is applied.
207+
// Whenever the feedbacks are updated, this method will be called.
208+
// The plugin can execute the rollout logic based on the feedback status changes.
209+
rpc ProgressRollout(RolloutPluginRequest) returns (google.protobuf.Empty);
210+
211+
// ValidateRolloutCompletion is called to validate the completion of the rollout.
212+
// It is used to check if the rollout is completed successfully.
213+
// If the validation is completed successfully, the plugin should return a SUCCEEDED result.
214+
// If the validation is still in progress, the plugin should return an INPROGRESS result.
215+
// If the validation fails, the plugin should return a FAILED result.
216+
rpc ValidateRolloutCompletion(ValidateCompletionRequest) returns (ValidateResponse);
217+
218+
// MutateManifestWork is called to mutate the manifestwork resource before it is applied or aborted.
219+
// MWRS Controller provides the current rollout status to the plugin.
220+
// The plugin can use this information to mutate the manifestwork resource.
221+
rpc MutateManifestWork(MutateManifestWorkRequest) returns (MutateManifestWorkResponse);
222+
}
223+
```
224+
225+
#### Request message
226+
227+
To address the user scenarios, the work controller will pass a common set of information in the gRPC request payload for each hook.
228+
229+
The plugin hook APIs will identify the ManifestWorkReplicaSet being processed with the following input:
230+
* MWRS Name: The name of the ManifestWorkReplicaSet resource.
231+
* MWRS Namespace: The namespace of the ManifestWorkReplicaSet resource.
232+
* Placement Name: The name of the Placement resource currently driving the rollout.
233+
* Current Cluster Name: The name of the specific managed cluster being progessed. only for hooks that operate on a specific cluster (e.g., MutateManifestWork, ValidateRolloutCompletion)
234+
* Total Cluster Count: The total number of clusters selected by the current Placement.
235+
* Rollout Status: The cluster rollout status (completed clusters, progressing clusters, timed_out clusters, removed clusters). Each entry in the list includes:
236+
- clusterName: The name of the cluster.
237+
- rolloutStatus: The current [cluster rollout status](https://github.com/open-cluster-management-io/sdk-go/blob/main/pkg/apis/cluster/v1alpha1/rollout.go#L23-L39) (e.g., ToApply, Progressing, Succeeded, Failed, TimeOut, Skip).
238+
- manifestRevisionName: The name of the manifest revision applied to the cluster.
239+
240+
#### Error handling
241+
242+
gRPC status codes follow the [standard gRPC status codes](https://grpc.github.io/grpc/core/md_doc_statuscodes.html): 0 = OK, 1 = CANCELLED, 2 = UNKNOWN, 3 = INVALID_ARGUMENT, 4 = DEADLINE_EXCEEDED, etc. Work controller will also utilize the [standard gRPC retry](https://grpc.io/docs/guides/retry/) for `UNAVAILABLE` status code.
243+
244+
Error messages from the plugin server are reported in status conditions:
245+
* When plugin initialization fails, the error message is shown in the `message` field of the `PluginLoaded` condition type.
246+
* When the plugin returns errors during rollout, the error message is shown in the `message` field of the `Progressing` condition type.
247+
248+
### Register custom plugins for work controller
249+
250+
A new `workConfiguration.plugins` field is introduced in ClusterManager to register the rollout plugins:
251+
252+
```yaml
253+
apiVersion: operator.open-cluster-management.io/v1
254+
kind: ClusterManager
255+
metadata:
256+
name: cluster-manager
257+
spec:
258+
...
259+
workImagePullSpec: quay.io/open-cluster-management/work:v1.0.0
260+
workConfiguration:
261+
workDriver: kube
262+
plugins:
263+
- name: my-rollout-1
264+
endpoint: my-rollout.my-namespace:10843
265+
caCertificate:
266+
caBundle: "REPLACE_WITH_BASE64_CA_CERT"
267+
- name: my-rollout-2
268+
endpoint: my-rollout.my-namespace:10843
269+
caCertificate:
270+
caBundle: "REPLACE_WITH_BASE64_CA_CERT"
271+
```
272+
273+
ClusterManager will create the following `work-controller-config` configmap in `open-cluster-management-hub` namespace.
274+
275+
```yaml
276+
apiVersion: v1
277+
kind: ConfigMap
278+
metadata:
279+
name: work-controller-config
280+
namespace: {{ .ClusterManagerNamespace }}
281+
data:
282+
config.yaml: |
283+
plugins:
284+
- name: my-rollout-1
285+
endpoint: my-rollout1.my-namespace:10843
286+
caCertificate:
287+
caBundle: "REPLACE_WITH_BASE64_CA_CERT"
288+
- name: my-rollout-2
289+
endpoint: my-rollout2.my-namespace:10843
290+
caCertificate:
291+
caBundle: "REPLACE_WITH_BASE64_CA_CERT"
292+
```
293+
294+
With the following helm chart change, work-controller will load work-controller-config configmap to get the plugin connection information.
295+
296+
```yaml
297+
apiVersion: apps/v1
298+
kind: Deployment
299+
metadata:
300+
name: {{ .ClusterManagerName }}-work-controller
301+
namespace: {{ .ClusterManagerNamespace }}
302+
spec:
303+
template:
304+
spec:
305+
...
306+
containers:
307+
- name: {{ .ClusterManagerName }}-work-controller
308+
image: {{ .WorkImage }}
309+
imagePullPolicy: IfNotPresent
310+
args:
311+
- "/work"
312+
- "manager"
313+
...
314+
{{ if .WorkConfigEnabled }}
315+
- "--work-config=/var/run/config/work/config.yaml"
316+
{{ end }}
317+
...
318+
volumeMounts:
319+
- name: tmpdir
320+
mountPath: /tmp
321+
...
322+
{{ if .WorkConfigEnabled }}
323+
- mountPath: /var/run/config/work
324+
name: workconfig
325+
readOnly: true
326+
{{ end }}
327+
volumes:
328+
...
329+
{{ if .WorkConfigEnabled }}
330+
- name: workconfig
331+
configMap:
332+
name: work-controller-config
333+
{{ end }}
334+
```
335+
336+
### Using plugin in MWRS
337+
338+
Plugin is an opt-in feature. User can use the registered plugin by setting `.spec.placementRefs[*].rolloutStrategy.plugin`. The reconciler makes sure that plugin is available and show its availability or error messages in `PluginLoaded` status condition type.
339+
340+
```yaml
341+
apiVersion: work.open-cluster-management.io/v1
342+
kind: ManifestWorkReplicaSet
343+
metadata:
344+
name: <MWRS_Name>
345+
spec:
346+
placementRefs:
347+
- name: placement-rollout-progressive
348+
rolloutStrategy:
349+
type: Progressive
350+
# plugin is optional.
351+
plugin: my-rollout
352+
progressive:
353+
...
354+
status:
355+
conditions:
356+
- lastTransitionTime: "2025-10-09T04:40:41Z"
357+
message: "my-rollout plugin is available."
358+
observedGeneration: 1
359+
reason: Available
360+
status: "True"
361+
type: PluginLoaded
362+
```
363+
364+
The following example shows `Available` and `Failed` reasons for `PluginLoaded` status condition type
365+
366+
| Reason | Status | Message |
367+
| --- | --- | --- |
368+
| Available | True | `my-rollout plugin is available.` |
369+
| Failed | False | `Initialize fails with unavaiable plugin server endpoint.` |
370+
371+
### Test Plan
372+
373+
- Unit-test
374+
- Integration-test: Create the sample K8s deployment based safe rollout plugin.
375+
376+
377+
### Upgrade / Downgrade Strategy
378+
379+
The plugin mechanism is opt-in. Work controller without plugin configuration behave identically to today’s work controller.
380+
381+
### Version Skew Strategy
382+
383+
The proposed changes introduce new condition and new fields in the exising custom resources.

0 commit comments

Comments
 (0)