Skip to content

benchmarks: avoid unreliable kubectl attach when fetching metrics#5296

Merged
p-se merged 2 commits into
mainfrom
fix-benchmarks
Jun 25, 2026
Merged

benchmarks: avoid unreliable kubectl attach when fetching metrics#5296
p-se merged 2 commits into
mainfrom
fix-benchmarks

Conversation

@p-se

@p-se p-se commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Fixes the benchmarks in conjunction with #5267

Additional Information

Checklist

  • I have updated the documentation via a pull request in the fleet-product-docs repository.

@p-se p-se requested a review from a team as a code owner June 17, 2026 06:24
Copilot AI review requested due to automatic review settings June 17, 2026 06:24
@p-se p-se changed the title Prevent collisions from pods checking metrics in benchmarks benchmarks: avoid unreliable kubectl attach when fetching metrics Jun 17, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adjusts how benchmark metrics are fetched to avoid output duplication/collisions caused by kubectl’s attach behavior, by switching to a “create pod → wait → read logs → delete pod” flow and by enhancing the e2e kubectl wrapper to expose stdout/stderr separately.

Changes:

  • Extend the e2e kubectl.Command wrapper to capture stdout and stderr independently, and add RunStdout.
  • Update benchmark metrics collection to avoid kubectl run --attach and instead fetch curl output via kubectl logs after the pod completes.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
e2e/testenv/kubectl/kubectl.go Split stdout/stderr capture; add RunStdout helper used by benchmarks.
benchmarks/record/record.go Replace kubectl run --attach --rm metrics fetch with pod lifecycle + kubectl logs approach.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread benchmarks/record/record.go Outdated
@p-se p-se force-pushed the fix-benchmarks branch from 30b87fb to b10bf28 Compare June 17, 2026 06:54
Comment thread benchmarks/record/record.go Outdated
return fmt.Errorf("curl pod %s finished with phase %s", pod, phase)
}

out, _, err := k.RunStdout("logs", "--namespace", "cattle-fleet-system", pod)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: in case we run into more troubles getting these logs, we could replace creating a new pod with a port forward for fetching metrics, as done in the dump CLI.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hope it won't be necessary, but yeah, we may have to do that.

Comment thread e2e/testenv/kubectl/kubectl.go
Comment thread e2e/testenv/kubectl/kubectl.go Outdated
@p-se p-se merged commit 67b7e5e into main Jun 25, 2026
24 checks passed
@p-se p-se deleted the fix-benchmarks branch June 25, 2026 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants