fix: implement thread-safe token caching for self-managed oidc

[steven-zou]

Description:

• Introduced a synchronized TokenCache struct with sync.Mutex in internal/authctx/client.go to manage the lifecycle of self-managed OIDC tokens.
• Updated TanzuContext to hold a pointer to TokenCache (smTokenCache), ensuring thread-safe access across multiple concurrent goroutines.
• Modified getUserAuthCtxHeaders to retrieve and refresh tokens through the cache, preventing duplicate authentication requests and race conditions when creating clusters in parallel.
• Updated getSMUserAuthCtx in internal/authctx/selfmanaged.go to return the token's expiration time (expiry time.Time).
• Updated refreshSMUserAuthCtx to explicitly update the cached token headers and expiry.

Testing Done:

Run terraform apply -auto-approve -parallelism=4 to create 4 clusters in parallel.
Race conditions error is reported:

╷
│ Error: Couldn't read TKG cluster.
│ Management Cluster Name: mgt-vc803ep7-01, Provisioner: testns, Cluster Name: cluster-001: error while setting auth headers: failed to exchange auth code for oauth tokens: Post "https://pinniped-supervisor.tmc.tanzu.io/provider/pinniped/oauth2/token": EOF
│
│   with tanzu-mission-control_tanzu_kubernetes_cluster.cluster-001,
│   on main.tf line 28, in resource "tanzu-mission-control_tanzu_kubernetes_cluster" "cluster-001":
│   28: resource "tanzu-mission-control_tanzu_kubernetes_cluster" "cluster-001" {
│
╵
╷
│ Error: Couldn't read TKG cluster.
│ Management Cluster Name: mgt-vc803ep7-01, Provisioner: testns, Cluster Name: cluster-002: error while setting auth headers: failed to exchange auth code for oauth tokens: Post "https://pinniped-supervisor.tmc.tanzu.io/provider/pinniped/oauth2/token": EOF
│
│   with tanzu-mission-control_tanzu_kubernetes_cluster.cluster-002,
│   on main.tf line 87, in resource "tanzu-mission-control_tanzu_kubernetes_cluster" "cluster-002":
│   87: resource "tanzu-mission-control_tanzu_kubernetes_cluster" "cluster-002" {
│
╵
╷
│ Error: Couldn't read TKG cluster.
│ Management Cluster Name: mgt-vc803ep7-01, Provisioner: testns, Cluster Name: cluster-003: error while setting auth headers: failed to initiate authorize request with issuer https://pinniped-supervisor.tmc.tanzu.io/provider/pinniped: authorization response error: Get "https://pinniped-supervisor.tmc.tanzu.io/provider/pinniped/oauth2/authorize?client_id=pinniped-cli&code_challenge=JzdE4YSOrTCL29FO5KhhbVm0GlnjzYRuqk5PaVcPkn0&code_challenge_method=S256&redirect_uri=http%3A%2F%2F127.0.0.1%2Fcallback&response_type=code&scope=openid+offline_access+username+groups&state=c5271ec01e5a1d1f06c2d331cde31e56": EOF
│
│   with tanzu-mission-control_tanzu_kubernetes_cluster.cluster-003,
│   on main.tf line 146, in resource "tanzu-mission-control_tanzu_kubernetes_cluster" "cluster-003":
│  146: resource "tanzu-mission-control_tanzu_kubernetes_cluster" "cluster-003" {
│
╵
╷
│ Error: Couldn't read TKG cluster.
│ Management Cluster Name: mgt-vc803ep7-01, Provisioner: testns, Cluster Name: cluster-004: error while setting auth headers: failed to initiate authorize request with issuer https://pinniped-supervisor.tmc.tanzu.io/provider/pinniped: authorization response error: Get "https://pinniped-supervisor.tmc.tanzu.io/provider/pinniped/oauth2/authorize?client_id=pinniped-cli&code_challenge=eUPG8WPRKYmNI0vorF8g4mL5mAuvFu6NVDlxSqn7rJ8&code_challenge_method=S256&redirect_uri=http%3A%2F%2F127.0.0.1%2Fcallback&response_type=code&scope=openid+offline_access+username+groups&state=304414a8df801e794a6ffac4bcf2f921": EOF
│
│   with tanzu-mission-control_tanzu_kubernetes_cluster.cluster-004,
│   on main.tf line 205, in resource "tanzu-mission-control_tanzu_kubernetes_cluster" "cluster-004":
│  205: resource "tanzu-mission-control_tanzu_kubernetes_cluster" "cluster-004" {

Run the same command with fixed version, no error reported.

Metadata:

Jira: TMC-69727

Made with AI.

[legal-compliance-bot]

🛑 Legal Compliance Check Failed

Hi @steven-zou, thank you for your contribution!

To merge this Pull Request, you must sign our CLA.

Note: Even if you signed off your commits locally (using git commit -s), you must post the comment below to register your signature with our automated system.
Note: This is a one-time process. Once signed, future contributions to this repository will be verified automatically.

1. Read the Document: Click here to read the CLA
2. Sign via Comment: Copy and paste the exact line below into a new comment on this Pull Request:

I have read the CLA Document and I hereby sign the CLA for this and all future contributions.

---

⏳ Processing Schedule:
Our 'Compliance Sweeper' runs automatically approximately every 15-20 minutes.
After you post the comment, your status will update automatically during the next scheduled run.
You do not need to take any further action.

[steven-zou]

I have read the CLA Document and I hereby sign the CLA for this and all future contributions.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.50%. Comparing base (3578fd2) to head (09cef0d).
⚠️ Report is 185 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/authctx/client.go	0.00%	31 Missing ⚠️
internal/authctx/selfmanaged.go	0.00%	17 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (3578fd2) and HEAD (09cef0d). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (3578fd2)	HEAD (09cef0d)
	2	1

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #677      +/-   ##
==========================================
- Coverage   25.23%   17.50%   -7.74%     
==========================================
  Files         195      264      +69     
  Lines       16908    17698     +790     
==========================================
- Hits         4267     3098    -1169     
- Misses      12423    14402    +1979     
+ Partials      218      198      -20     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tenthirtyam]

LGTM, @steven-zou! TY!