Summary
Add an opt-in async output artifact upload mode.
Today, a workflow task does not become available to unrelated downstream work until its output artifact uploads finish. For large artifacts, this can unnecessarily block DAG or steps branches that do not consume those artifacts.
The proposal is to let Argo report output metadata, including final artifact locations, as soon as those locations are known, then continue uploading artifact bytes in the wait container. Tasks that consume those artifacts should still wait until the referenced upload has completed successfully. Workflow completion and pod cleanup should also wait for pending async uploads by default.
Use Cases
Our main use case is around build cache; allowing workflow to continue and upload the cache in the background can save precious minutes in case of large builds (Rust, big UI node projects).
Message from the maintainers:
Love this feature request? Give it a 👍. We prioritise the proposals with the most 👍.
Summary
Add an opt-in async output artifact upload mode.
Today, a workflow task does not become available to unrelated downstream work until its output artifact uploads finish. For large artifacts, this can unnecessarily block DAG or steps branches that do not consume those artifacts.
The proposal is to let Argo report output metadata, including final artifact locations, as soon as those locations are known, then continue uploading artifact bytes in the wait container. Tasks that consume those artifacts should still wait until the referenced upload has completed successfully. Workflow completion and pod cleanup should also wait for pending async uploads by default.
Use Cases
Our main use case is around build cache; allowing workflow to continue and upload the cache in the background can save precious minutes in case of large builds (Rust, big UI node projects).
Message from the maintainers:
Love this feature request? Give it a 👍. We prioritise the proposals with the most 👍.