fix: Add explicit HTTP request timeouts to all AWS SDK clients by howard-junec · Pull Request #1374 · kubernetes/cloud-provider-aws

howard-junec · 2026-04-06T19:10:48Z

Add explicit HTTP request timeouts (30s) to all AWS SDK clients using the SDK's BuildableClient via WithHTTPClient on every LoadDefaultConfig call. Using BuildableClient instead of a bare http.Client preserves the SDK's default transport settings (TLS 1.2, connection pooling, dialer config). This prevents the Go SDK's clock skew overcorrection from breaking all API calls when a single response is slow. Added AST test to fail if any future LoadDefaultConfig is added without WithHTTPClient.

/kind bug

What this PR does / why we need it:
Add explicit HTTP request timeouts (30s) to all AWS SDK clients by configuring
WithHTTPClient on every LoadDefaultConfig call. Without a timeout, a single
slow API response can trigger the Go AWS SDK v2's clock skew overcorrection,
breaking all subsequent API calls with HTTP 401 errors.

Also adds an AST-based test that scans source files and fails if any
LoadDefaultConfig is added without WithHTTPClient.

Which issue(s) this PR fixes:
NONE

Special notes for your reviewer:
tests/e2e/loadbalancer.go has one LoadDefaultConfig without timeout —
intentionally excluded since it's test tooling, not production controller code.
The AST test skips the tests/ directory accordingly.

Does this PR introduce a user-facing change?:

NONE

Configure WithHTTPClient on all LoadDefaultConfig calls to prevent clock skew overcorrection from slow API responses (COE-389792). Add AST test to ensure future SDK clients include timeout.

linux-foundation-easycla · 2026-04-06T19:10:55Z

The committers listed above are authorized under a signed CLA.

✅ login: howard-junec / name: junhe cao (755f116, 8cdaecc)

k8s-ci-robot · 2026-04-06T19:10:57Z

This issue is currently awaiting triage.

If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2026-04-06T19:10:57Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign dims for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2026-04-06T19:10:58Z

Welcome @howard-junec!

It looks like this is your first PR to kubernetes/cloud-provider-aws 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/cloud-provider-aws has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2026-04-06T19:10:58Z

Hi @howard-junec. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

kmala · 2026-04-06T22:35:29Z

cmd/ecr-credential-provider/main.go

 	if region != "" {
 		cfg, err = config.LoadDefaultConfig(ctx,
 			config.WithRegion(region),
+			config.WithHTTPClient(defaultHTTPClient),


Can we just override the the default client's timeout since aws sdk has other default for transport which are good to take https://github.com/aws/aws-sdk-go-v2/blob/main/aws/transport/http/client.go#L44

thanks, I will update to use awshttp.NewBuildableClient().WithTimeout(30 * time.Second) to preserve the SDK's default transport settings.

kmala · 2026-04-06T22:38:18Z

pkg/providers/v1/aws_sdk.go

+// HTTP request timeout and reuse connection pools. Without a timeout, a single
+// slow response can trigger the Go SDK's clock skew overcorrection and break all
+// subsequent API calls (COE-389792).
+var defaultHTTPClient = &http.Client{Timeout: 30 * time.Second}


is 30 sec enough to list all the instances or does pagination take care of that ?

The 30s timeout is per HTTP request, not per operation. DescribeInstances paginates via NextToken, each page is a separate HTTP call with its own 30s window.

Replace http.Client with awshttp.NewBuildableClient().WithTimeout() to preserve SDK default transport settings.

kmala · 2026-04-08T01:22:34Z

/ok-to-test
/lgtm

k8s-ci-robot · 2026-04-08T01:22:50Z

@howard-junec: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cloud-provider-aws-check	`8cdaecc`	link	true	`/test pull-cloud-provider-aws-check`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

fix: Add explicit HTTP request timeouts to all AWS SDK clients

755f116

Configure WithHTTPClient on all LoadDefaultConfig calls to prevent clock skew overcorrection from slow API responses (COE-389792). Add AST test to ensure future SDK clients include timeout.

k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Apr 6, 2026

k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Apr 6, 2026

k8s-ci-robot requested review from JoelSpeed and yue9944882 April 6, 2026 19:10

kmala reviewed Apr 6, 2026

View reviewed changes

Use SDK BuildableClient for HTTP timeouts

8cdaecc

Replace http.Client with awshttp.NewBuildableClient().WithTimeout() to preserve SDK default transport settings.

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 8, 2026

k8s-ci-robot assigned kmala Apr 8, 2026

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 8, 2026

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Apr 8, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Add explicit HTTP request timeouts to all AWS SDK clients#1374

fix: Add explicit HTTP request timeouts to all AWS SDK clients#1374
howard-junec wants to merge 2 commits intokubernetes:masterfrom
howard-junec:add-aws-sdk-http-timeout

howard-junec commented Apr 6, 2026 •

edited by kmala

Loading

Uh oh!

linux-foundation-easycla bot commented Apr 6, 2026 •

edited

Loading

Uh oh!

k8s-ci-robot commented Apr 6, 2026

Uh oh!

k8s-ci-robot commented Apr 6, 2026

Uh oh!

k8s-ci-robot commented Apr 6, 2026

Uh oh!

k8s-ci-robot commented Apr 6, 2026

Uh oh!

kmala Apr 6, 2026

Uh oh!

howard-junec Apr 7, 2026 •

edited

Loading

Uh oh!

kmala Apr 6, 2026

Uh oh!

howard-junec Apr 7, 2026

Uh oh!

kmala commented Apr 8, 2026

Uh oh!

k8s-ci-robot commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

howard-junec commented Apr 6, 2026 • edited by kmala Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linux-foundation-easycla bot commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Apr 6, 2026

Uh oh!

k8s-ci-robot commented Apr 6, 2026

Uh oh!

k8s-ci-robot commented Apr 6, 2026

Uh oh!

k8s-ci-robot commented Apr 6, 2026

Uh oh!

kmala Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

howard-junec Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kmala Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

howard-junec Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

kmala commented Apr 8, 2026

Uh oh!

k8s-ci-robot commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

howard-junec commented Apr 6, 2026 •

edited by kmala

Loading

linux-foundation-easycla bot commented Apr 6, 2026 •

edited

Loading

howard-junec Apr 7, 2026 •

edited

Loading