feat: pause auto-promotion after promoting older freight by jacobboykin · Pull Request #6334 · akuity/kargo

jacobboykin · 2026-05-22T03:06:54Z

Relates to #3016

Summary

This adds auto-promotion holds for Stages.

When a Promotion other than the Stage controller's own auto-promotion succeeds for Freight that is not the current auto-promotion candidate for the same origin, Kargo records a hold that pauses auto-promotion for that origin on that Stage. That keeps later auto-promotion from moving the Stage back to the candidate. Other origins on the same Stage can continue to auto-promote.

Auto-promotion resumes after a Promotion other than the Stage controller's own auto-promotion succeeds for the current candidate for that origin.

How It Works

For created Promotions on auto-promotion-enabled Stages, the Promotion webhook infers promotion intent unless the Promotion is marked as one created by the Stage controller's auto-promotion loop. That keeps Kargo's own auto-promotions from creating holds while allowing UI, CLI, kubectl, external automation, and Enterprise auto-rollback Promotions to participate without special handling.

If the selected Freight is not the current auto-promotion candidate, the webhook stamps a hold intent annotation.
If the selected Freight is the current candidate, the webhook stamps a release intent annotation.
Stage-controller auto-promotions are not stamped.

The Stage controller is the only writer of committed hold state. It records active holds in status.autoPromotionHolds, keyed by origin, after a hold-intent Promotion succeeds. A succeeded release-intent Promotion clears the hold.

Hold processing uses a timestamp/name watermark so effects are applied once and remain stable even after Promotion GC.

autoPromoteFreight skips an origin when:

an active hold exists for that origin, or
a non-terminal hold-intent Promotion is still pending or running.

That second check closes the window where a user-selected hold Promotion exists but has not yet reached Stage status.

Promote By Origin

This also adds a small general primitive: promote by origin.

REST, CLI, and raw Promotion specs can now ask Kargo to promote the current candidate for an origin in the direct Stage promotion path:

kargo promote \
  --project my-project \
  --stage test \
  --origin Warehouse/my-warehouse

For raw Kubernetes usage, create a Promotion with spec.origin instead of spec.freight. The mutating webhook resolves the origin to the current candidate and fills spec.freight before the Promotion is persisted.

The REST promote request now accepts exactly one of freight, freightAlias, or origin. The Promotion CRD accepts exactly one of spec.freight or spec.origin.

ConnectRPC is intentionally left as-is because it is deprecated and has no current caller for promote-by-origin.

UI And Docs

The UI now surfaces active holds from Stage.status.autoPromotionHolds:

Stage nodes show auto-promotion as paused when holds exist.
The current Freight view marks held origins.
Stage details list the held origin, Freight, Promotion, actor, and timestamp when present.

This PR does not add a dedicated resume button or pending-hold UI. Resume is already supported by promoting the current candidate, including with --origin.

Docs were updated to explain when auto-promotion pauses and how to resume it.

netlify · 2026-05-22T03:06:58Z

✅ Deploy Preview for docs-kargo-io ready!

Name	Link
🔨 Latest commit	`884d325`
🔍 Latest deploy log	https://app.netlify.com/projects/docs-kargo-io/deploys/6a501acd5d97ea00081e4857
😎 Deploy Preview	https://deploy-preview-6334.docs.kargo.io
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

codecov · 2026-06-03T22:10:36Z

Codecov Report

❌ Patch coverage is 71.77700% with 162 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.08%. Comparing base (9aeaadd) to head (884d325).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/cli/cmd/promote/promote.go	0.00%	39 Missing and 1 partial ⚠️
pkg/controller/stages/regular_stages.go	81.02%	27 Missing and 10 partials ⚠️
pkg/webhook/kubernetes/promotion/webhook.go	85.13%	17 Missing and 5 partials ⚠️
pkg/server/promote_to_stage_v1alpha1.go	72.72%	12 Missing and 6 partials ⚠️
cmd/controlplane/kubernetes_webhooks.go	0.00%	16 Missing ⚠️
pkg/api/auto_promotion.go	85.50%	5 Missing and 5 partials ⚠️
pkg/cli/cmd/get/freight.go	0.00%	9 Missing ⚠️
pkg/api/annotations.go	0.00%	4 Missing ⚠️
pkg/cli/option/flag.go	0.00%	4 Missing ⚠️
pkg/server/promote_downstream_v1alpha1.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6334      +/-   ##
==========================================
+ Coverage   58.77%   59.08%   +0.31%     
==========================================
  Files         506      507       +1     
  Lines       42928    43278     +350     
==========================================
+ Hits        25232    25572     +340     
+ Misses      16157    16151       -6     
- Partials     1539     1555      +16

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Marvin9 · 2026-06-19T06:13:35Z

I'll leave the final decision to @Marvin9, since he's currently working on the migration.

Yes please wait for that PR to merge. I will help with any UI conflict resolution in this PR, would be much easier than doing it in that PR

krancour · 2026-06-23T17:07:09Z

Ok... I've obviously put a lot of effort into understanding the choices in this PR. I realize that my many in-line comments don't tell the whole story, as I see it. So, please just feel free to dismiss all of those. This one comment is a comprehensive snapshot of my thinking here. (I have, in fact, marked all my other comments as resolved to reduce overall noise.)

First, credit where it's due: generally speaking, I think the contours of the proposed changes are logically correct. My concern isn't correctness; it's surface area. I think a few foundational choices are each pulling in a lot of machinery, and that removing them collapses most of the complexity without losing the behavior we actually want.

What I think is driving the complexity

New and changed API endpoints. A candidate-fetch endpoint, a resume endpoint, and new fields on both the gRPC and REST promote calls. This also means the feature only works for UI/CLI users, not kubectl.
ExpectedAutoCandidate + the candidate endpoint. This guards a race by narrowing it, not closing it — and the endpoint exists almost entirely to feed it. I don't think the race justifies the cost.
The shared createStagePromotion wrapper, which forced a third error vocabulary (stagePromotionConflictError and its translators) and the protocol-named getRESTStage helper.
The Promotion.Source enum + the Promotion-controller hard gate + hold-first status writes from the API server — two writers of hold state, hence the heavy optimistic locking.

The leaner shape I'd like us to consider

No API changes at all — no new endpoints, no changes to the gRPC or REST promote calls. Move all intent inference into the Promotion defaulting/mutating webhook on create. The enhancement then applies uniformly to UI, CLI, and kubectl.
The webhook stamps intent as annotations. It computes the auto-promotion candidate for the promoted Freight's origin: promoting something older than the candidate → a "rollback" annotation (intent to hold) carrying the origin; promoting the candidate → a "clear" annotation (intent to release).
The Stage controller is solely responsible for establishing and releasing holds in accordance with promotion intent.
autoPromoteFreight skips an origin that has a status hold or a non-terminal rollback-annotated Promotion. The second condition closes the only window worth closing, cheaply.
Release stays implicit — manually promoting the auto-promotion candidate clears the hold. No manual resume endpoint required.

Single writer means we can drop the optimistic-locking apparatus, the gate, Source, ExpectedAutoCandidate, both new endpoints, the wrapper, and the bespoke error types.

Behavior is unchanged from a user's perspective: manually promoting older Freight pauses auto-promotion for that origin, promoting the current candidate resumes it, and pauses are per-origin.

Two residual races remain, and I'm comfortable accepting both with a clear comment in the webhook: a benign, self-clearing intent misclassification, and a transient (self-correcting, because the queue runs oldest-first) case where newer Freight briefly promotes before the rollback lands. Neither is worth more machinery.

Happy to talk it through.

Signed-off-by: fuskovic <fhuskovic92@gmail.com>

…otions The Promotion webhook skipped hold/release intent inference for any admission request originating from the Kargo control plane, on the theory that such Promotions are system-generated and carry no user intent. But the Kargo API server is part of the control plane and creates Promotions on users' behalf using its own service account -- so Promotions created via the UI, CLI, or REST API (the primary user flows) never received intent annotations, and manual promotions made through them could neither establish nor release auto-promotion holds. The feature effectively worked only for Promotions created directly with kubectl. The discriminator for "does this control-plane Promotion carry user intent" already exists: the API server records the requesting user in the create-actor annotation before creating a Promotion on their behalf. The Stage controller's own auto-promotions set no create-actor at all, and other controllers (e.g. a rollback controller) identify themselves with a controller: actor. The webhook now infers intent for control-plane requests whose Promotion names a non-controller actor, and continues to leave genuinely system-generated Promotions untouched -- including preserving any intent annotations they set explicitly. Note that matching a user actor by prefix is not possible: the create-actor value for users varies by identity ("admin", "email:...", "subject:...", claim-based forms), so the rule keys on the absence of an actor or the controller: prefix instead. Signed-off-by: Kent Rancourt <kent.rancourt@gmail.com>

…ets set Signed-off-by: fuskovic <fhuskovic92@gmail.com>

Signed-off-by: fuskovic <fhuskovic92@gmail.com>

…er-wins patch status race Signed-off-by: fuskovic <fhuskovic92@gmail.com>

Reverts 2320ce0 and 7d2be5f. The state the first guarded against -- a terminal Promotion that was never acknowledged as current -- is unreachable: the promotions controller will not run a Promotion before the Stage records it as current. The race actually observed is elsewhere: a Promotion can go terminal between syncPromotions computing hold state and autoPromoteFreight acting on it. That will be fixed by having the auto-promotion guard also consider terminal hold-intent Promotions newer than status.lastPromotion. The second added optimistic locking to PatchStatus, which every status writer in the codebase uses. That is too large a behavioral change to adopt as a side effect of this PR -- and it would not have prevented the harmful outcome here, which is a Promotion creation, not a status write. Signed-off-by: Kent Rancourt <kent.rancourt@gmail.com>

A Promotion can reach a terminal phase in the interval between syncPromotions computing hold state and autoPromoteFreight acting on it. The guard for in-flight hold intent only considered non-terminal Promotions, so one that succeeded moments earlier -- its hold not yet recorded -- did not block, and auto-promotion could supersede older Freight the user just deliberately promoted. The guard now also blocks on terminal hold-intent Promotions newer than status.lastPromotion, i.e., those whose outcome has not been recorded yet. Once the next reconciliation records the outcome, the hold itself takes over the blocking (or, for a failed Promotion, auto-promotion correctly resumes). Also documents the mirror ordering of accepted race 2 in the Promotion webhook: a queued auto-promotion created moments before a hold-intent Promotion may supersede it once; the hold survives (auto-promotions never imply intent) and recovery is re-promoting the older Freight. No read-time guard can see a Promotion created after the read. Signed-off-by: Kent Rancourt <kent.rancourt@gmail.com>

Signed-off-by: fuskovic <fhuskovic92@gmail.com>

…andidate Same window as the previous commit, different consumer: a fast Promotion can go from pending to succeeded between syncPromotions observing it and autoPromoteFreight acting. The duplicate-suppression check only matched non-terminal Promotions, and a succeeded Promotion for the candidate is deliberately not disqualifying (holds replaced the old any-Promotion-exists guard, which had been suppressing this duplicate incidentally). Auto-promotion could therefore create a second, identical Promotion moments after the first succeeded. The check now stands down while any Promotion for the candidate is either non-terminal or terminal but newer than status.lastPromotion. Once the outcome is recorded, a success is caught by the candidate-is-already-current check and a failure by the newest-terminal-not-successful check, so recorded history still never blocks -- the old guard stays removed. Signed-off-by: Kent Rancourt <kent.rancourt@gmail.com>

controller-runtime demotes any untyped error returned from a defaulter to a generic admission denial, which kube-apiserver surfaces as HTTP 403 Forbidden. This made every failure in the Promotion mutating webhook -- including 'no auto-promotion candidate found' during promote-by-origin and transient freight-listing failures -- masquerade as an authorization problem. Return apierrors.NewInvalid (422) for user-correctable conditions (missing Stage, Stage with no promotion steps, no candidate for origin) and apierrors.NewInternalError (500) for infrastructure failures, matching the idiom the validating webhook already uses. Step inflation failures now pass through any underlying typed status error instead of being flattened. Signed-off-by: Kent Rancourt <kent.rancourt@gmail.com>

The auto-promotion and hold documentation spoke loosely of pausing auto-promotion 'for an origin' without ever using the nomenclature Stages actually use: freight requests (the entries of spec.requestedFreight). Introduce the term where requestedFreight is first described, note that requests behave like slots each holding one piece of Freight from a specific origin, and define the per-request auto-promotion candidate. Holds are now described precisely as established on, recorded for, and lifted from a specific freight request, leaving the Stage's other freight requests unaffected. Also document that hold/resume intent takes effect only when a Promotion succeeds and that auto-promotion's own Promotions never carry intent. Signed-off-by: Kent Rancourt <kent.rancourt@gmail.com>

Signed-off-by: fuskovic <fhuskovic92@gmail.com>

krancour · 2026-07-10T01:10:41Z

+		{field: indexer.FreightByWarehouseField, fn: indexer.FreightByWarehouse},
+		{field: indexer.FreightByCurrentStagesField, fn: indexer.FreightByCurrentStages},
+		{field: indexer.FreightByVerifiedStagesField, fn: indexer.FreightByVerifiedStages},
+		{field: indexer.FreightApprovedForStagesField, fn: indexer.FreightApprovedForStages},


@fuskovic, non-blocking, but it's worth checking that after all of the refactoring, all of these are actually still used.

kargo-governance-bot Bot assigned jacobboykin May 22, 2026

jacobboykin changed the title ~~Pause auto-promotion after rollback~~ feat: pause auto-promotion after rollback May 22, 2026

jacobboykin force-pushed the jacobboykin/stage-pinning-auto-promotion-holds branch from 1491789 to 18a0ed0 Compare May 22, 2026 03:28

jacobboykin force-pushed the jacobboykin/stage-pinning-auto-promotion-holds branch from c60a74e to 19e84b8 Compare June 3, 2026 22:02

jessesuen reviewed Jun 4, 2026

View reviewed changes

Comment thread pkg/api/auto_promotion.go Outdated

hairyhum reviewed Jun 5, 2026

View reviewed changes

Comment thread api/v1alpha1/stage_types.go Outdated

jacobboykin force-pushed the jacobboykin/stage-pinning-auto-promotion-holds branch 2 times, most recently from 1d5dd27 to 76aa755 Compare June 11, 2026 01:52

jacobboykin commented Jun 11, 2026

View reviewed changes

Comment thread pkg/api/auto_promotion.go Outdated

jacobboykin marked this pull request as ready for review June 11, 2026 14:54

jacobboykin requested review from a team as code owners June 11, 2026 14:54

jacobboykin requested review from hairyhum, jessesuen, rpelczar and thomastaylor312 June 11, 2026 15:02

Marvin9 reviewed Jun 16, 2026

View reviewed changes

Comment thread ui/src/features/project/pipelines/nodes/promotion/use-get-promotion-dropdown-items.tsx Outdated

Marvin9 previously requested changes Jun 16, 2026

View reviewed changes

Comment thread ui/src/features/project/pipelines/nodes/promotion/use-get-promotion-dropdown-items.tsx Outdated

This comment was marked as outdated.

Sign in to view

krancour reviewed Jun 18, 2026

View reviewed changes

krancour reviewed Jun 22, 2026

View reviewed changes

Comment thread pkg/server/promote_to_stage_v1alpha1.go Outdated

krancour reviewed Jun 22, 2026

View reviewed changes

Comment thread pkg/server/auto_promotion_v1alpha1.go Outdated

krancour reviewed Jun 22, 2026

View reviewed changes

Comment thread pkg/server/promote_to_stage_v1alpha1.go Outdated

krancour reviewed Jul 7, 2026

View reviewed changes

Comment thread pkg/cli/cmd/promote/promote.go Outdated

krancour reviewed Jul 7, 2026

View reviewed changes

Comment thread pkg/controller/stages/regular_stages.go Outdated

fuskovic added 3 commits July 8, 2026 03:31

fix latest freight language

8662bc0

Signed-off-by: fuskovic <fhuskovic92@gmail.com>

codegen

5b35258

Signed-off-by: fuskovic <fhuskovic92@gmail.com>

drop unreachable code path

032aae5

Signed-off-by: fuskovic <fhuskovic92@gmail.com>

kamalmarhubi mentioned this pull request Jul 8, 2026

Generalise promotion holds to cover more use cases #6567

Open

2 tasks

fuskovic added 2 commits July 8, 2026 18:58

add more reconciler tests

11b5740

Signed-off-by: fuskovic <fhuskovic92@gmail.com>

add more webhook tests

116d895

Signed-off-by: fuskovic <fhuskovic92@gmail.com>

krancour reviewed Jul 8, 2026

View reviewed changes

Comment thread pkg/webhook/kubernetes/promotion/webhook.go

krancour reviewed Jul 8, 2026

View reviewed changes

Comment thread pkg/controller/stages/regular_stages.go Outdated

krancour reviewed Jul 8, 2026

View reviewed changes

Comment thread pkg/controller/stages/regular_stages.go Outdated

fuskovic and others added 3 commits July 8, 2026 23:34

fix stale comments

32af381

Signed-off-by: fuskovic <fhuskovic92@gmail.com>

Merge branch 'main' into jacobboykin/stage-pinning-auto-promotion-holds

bcca7af

krancour force-pushed the jacobboykin/stage-pinning-auto-promotion-holds branch from efe8eb0 to 184679c Compare July 8, 2026 21:24

fuskovic added 3 commits July 9, 2026 16:18

fix race condition for fast promos that finish before current promo g…

2320ce0

…ets set Signed-off-by: fuskovic <fhuskovic92@gmail.com>

divide test paths by auto-promo-selection policies

f787b27

Signed-off-by: fuskovic <fhuskovic92@gmail.com>

lint

755cae5

Signed-off-by: fuskovic <fhuskovic92@gmail.com>

fuskovic mentioned this pull request Jul 9, 2026

fix: make PatchStatus conflict-safe via optimistic locking #6581

Closed

fuskovic and others added 9 commits July 9, 2026 18:15

fix scenario E1 test for newestFreight policy path e.g. fix last-writ…

7d2be5f

…er-wins patch status race Signed-off-by: fuskovic <fhuskovic92@gmail.com>

add script for automating running newestFreight e2e tests

bc56615

Signed-off-by: fuskovic <fhuskovic92@gmail.com>

add script for automating matchesUpstream scenario e2e tests

0d22153

Signed-off-by: fuskovic <fhuskovic92@gmail.com>

remove hack/testing/auto-promotion-hold

884d325

Signed-off-by: fuskovic <fhuskovic92@gmail.com>

krancour mentioned this pull request Jul 10, 2026

fix: unblock auto-promotion with required soak times #6582

Draft

22 tasks

krancour reviewed Jul 10, 2026

View reviewed changes

Uh oh!

Conversation

jacobboykin commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How It Works

Promote By Origin

UI And Docs

Uh oh!

netlify Bot commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for docs-kargo-io ready!

Uh oh!

codecov Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Marvin9 commented Jun 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

krancour commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What I think is driving the complexity

The leaner shape I'd like us to consider

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

krancour Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

jacobboykin commented May 22, 2026 •

edited

Loading

netlify Bot commented May 22, 2026 •

edited

Loading

codecov Bot commented Jun 3, 2026 •

edited

Loading

krancour commented Jun 23, 2026 •

edited

Loading