Skip to content

feat(executor): report artifact metadata before upload completes#16104

Open
SimbaKingjoe wants to merge 1 commit into
argoproj:mainfrom
SimbaKingjoe:feat/async-artifact-upload
Open

feat(executor): report artifact metadata before upload completes#16104
SimbaKingjoe wants to merge 1 commit into
argoproj:mainfrom
SimbaKingjoe:feat/async-artifact-upload

Conversation

@SimbaKingjoe
Copy link
Copy Markdown

Summary

Report artifact metadata (S3 keys, types) to the controller before the actual upload completes. The wait container generates artifact keys from the deterministic archive location, reports outputs immediately, then uploads files concurrently via goroutines. This allows the controller to access artifact metadata earlier, and makes uploads non-blocking within the wait container.

Fixes

#16091

Motivation

Today, SaveArtifacts uploads all output artifacts synchronously before ReportOutputs. For large artifacts (build caches, ML models), this blocks the wait container and prevents the controller from seeing any output metadata until every byte is uploaded. For downstream DAG/Steps tasks that do NOT consume these artifacts, this is unnecessary delay.

This PR lays the foundation for full async artifact uploads by:

  1. Decoupling key generation from file upload
  2. Reporting metadata early
  3. Making uploads concurrent within the container

Modifications

workflow/executor/executor.go (+111/-13):

  • generateArtifactKey() — new method that computes the artifact's S3/GCS key from the archive location, without uploading
  • StagedArtifact — new struct pairing an artifact with its staged local file path
  • GenerateArtifactOutputs() — stages artifact files and populates keys/types, returns []StagedArtifact
  • SaveArtifactsAsync() — uploads staged artifacts concurrently via goroutines with sync.WaitGroup + error aggregation
  • saveArtifactFromFile() — refactored to call generateArtifactKey instead of inlining key generation

cmd/argoexec/commands/wait.go (+21/-1):

  • Reordered wait container flow: GenerateArtifactOutputsReportOutputsSaveArtifactsAsync
  • Metadata reported immediately, uploads run concurrently after

workflow/executor/executor_test.go (+47):

  • TestGenerateArtifactKey — verifies key generation and HasKey() idempotency
  • TestGenerateArtifactOutputs_EmptyOutputs — empty outputs return nil
  • TestSaveArtifactsAsync_EmptyList — nil/empty input returns nil

Verification

go test ./workflow/executor/ -short    # all existing + new tests pass
go build ./workflow/executor/           # compiles
go build ./cmd/argoexec/commands/       # compiles

Behavioral Impact

  • Backward compatible: FinalizeOutput (which sets report-outputs-completed=true) is still deferred and runs after all uploads complete. Downstream DAG dependencies still wait correctly.
  • Metadata available earlier: Controller's taskResultReconciliation copies outputs to NodeStatus regardless of completion flag (taskresult.go:124-143), so artifact keys are visible in UI/CLI as soon as ReportOutputs completes.
  • Concurrent uploads: Multiple artifacts now upload in parallel goroutines instead of serial.

@SimbaKingjoe SimbaKingjoe force-pushed the feat/async-artifact-upload branch from f68cde8 to 4902409 Compare May 17, 2026 13:01
Generate artifact keys (S3/GCS/etc) before uploading, then report
outputs with full metadata to the controller immediately. Actual
artifact uploads run concurrently via goroutines with WaitGroup
synchronization.

- Extract key generation from saveArtifactFromFile into generateArtifactKey
- Add StagedArtifact type pairing artifact metadata with local file paths
- Add GenerateArtifactOutputs to stage files and generate keys without uploading
- Add SaveArtifactsAsync to upload staged artifacts concurrently
- Update wait container flow: GenerateArtifactOutputs \u2192 ReportOutputs \u2192 SaveArtifactsAsync
- Add unit tests for generateArtifactKey, GenerateArtifactOutputs, SaveArtifactsAsync

Refs argoproj#16091

Signed-off-by: daixin1204 <daixin1204@gmail.com>
@SimbaKingjoe SimbaKingjoe force-pushed the feat/async-artifact-upload branch from 4902409 to 8156a36 Compare May 17, 2026 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant