Karpenter can provision EC2 instance types unsupported by EKS VPC Resource Controller (SG for Pods), causing pods to remain Pending with Insufficient vpc.amazonaws.com/pod-eni

[kamal-kailasababu]

Description

Observed Behavior:
We run EKS with Security Groups for Pods enabled (ENABLE_POD_ENI=true, enforcing mode standard). During scheduling, Karpenter provisioned a new node with instance type r8a.xlarge. The node then emitted an event:

Unsupported: The instance type r8a.xlarge is not supported yet by the vpc resource controller

As a result, pods requiring Pod ENIs could not be scheduled and remained Pending with:

FailedScheduling: Insufficient vpc.amazonaws.com/pod-eni

Karpenter provisions nodes (e.g. r8a.xlarge) that later cannot be trunk-enabled by the EKS-managed VPC Resource Controller. Pods requiring vpc.amazonaws.com/pod-eni remain Pending with Insufficient vpc.amazonaws.com/pod-eni.

Expected Behavior:
Karpenter identify the EKS VPC Resource Controller version of current cluster and won't provision a node which is not in the list - https://github.com/aws/amazon-vpc-resource-controller-k8s/blob/v1.7.15/pkg/aws/vpc/limits.go

Ideally, Karpenter would help avoid provisioning instance types that are incompatible with SG-for-Pods ENI trunking in the current control plane, or at least provide a clearer/first-class mechanism to prevent this mismatch.

Reproduction Steps (Please include YAML):

1. EKS cluster with Security Groups for Pods enabled:

   • ENABLE_POD_ENI=true
   • POD_SECURITY_GROUP_ENFORCING_MODE=standard

   Command:
   kubectl -n kube-system describe ds aws-node | grep -E 'ENABLE_POD_ENI|POD_SECURITY_GROUP_ENFORCING_MODE'

2. Karpenter NodePool allows the new family (example: r8a is not excluded).

3. Deploy a workload that requires Pod ENIs (Security Groups for Pods).

4. Karpenter provisions r8a.xlarge which is not in the EKS VPC Resource Controller version 1.7.15

5. Node event shows VPC RC incompatibility:
   Unsupported: The instance type r8a.xlarge is not supported yet by the vpc resource controller

6. Pod remains pending with:
   FailedScheduling: Insufficient vpc.amazonaws.com/pod-eni

Evidence / references

• EKS Security Groups for Pods docs:
  https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html

• Node event reporting controller version:
  ControllerVersionNotice: The node is managed by VPC resource controller version v1.7.15

• Node event reporting incompatibility:
  Unsupported: The instance type r8a.xlarge is not supported yet by the vpc resource controller

• VPC resource controller compatibility list (limits.go):
  https://github.com/aws/amazon-vpc-resource-controller-k8s/blob/v1.7.15/pkg/aws/vpc/limits.go

Mitigation

We mitigated by excluding r8a from our Karpenter NodePool requirements so Karpenter cannot provision that family for SG-for-Pods workloads.

Versions:

• Chart Version: 1.8.5
• Kubernetes Version (kubectl version): 1.34

Suggested here to re-open in this repo - kubernetes-sigs/karpenter#2863 (comment)

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[ryan-mist]

Karpenter also maintains a VPC resource list, the same as the VPC resource controllers one - https://github.com/aws/karpenter-provider-aws/blob/main/pkg/providers/instancetype/zz_generated.vpclimits.go

When vpc.amazonaws.com/pod-eni is requested, we do check to make sure its trunk-enabled - https://github.com/aws/karpenter-provider-aws/blob/main/pkg/providers/instancetype/types.go#L396-L403

func awsPodENI(instanceTypeName string) *resource.Quantity {
	// https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html#supported-instance-types
	limits, ok := Limits[instanceTypeName]
	if ok && limits.IsTrunkingCompatible {
		return resources.Quantity(fmt.Sprint(limits.BranchInterface))
	}
	return resources.Quantity("0")
}

Karpenter v1.8.5 had r8a.xlarge in the list, but the VPC resource Controller was an older version which did not include it.

---

Karpenter identify the EKS VPC Resource Controller version of current cluster and won't provision a node which is not in the list

It is possible to surface a configuration for this, but it does means Karpenter must maintain config for each version of the VPC resource controller and adds explicit version compatibility between the two.

Was there anything preventing an upgrade to a newer VPC resource controller version? I wonder if the easier way to solve this is ensuring that the VPC resource controller is upgraded before upgrading to a newer Karpenter version

[kamal-kailasababu]

@ryan-mist , Thanks for the details — that helps clarify the behavior on the Karpenter side.

Karpenter v1.8.5 had r8a.xlarge in the list, but the VPC resource Controller was an older version which did not include it.

That’s exactly the gap we’re running into: Karpenter’s trunking compatibility list can move ahead of the EKS-managed VPC Resource Controller version running in the control plane. When AWS introduces a new instance family (like r8a) and Karpenter includes it in zz_generated.vpclimits.go, Karpenter may legitimately consider it trunking-compatible. But if the control plane VPC Resource Controller in our cluster is still on an older version that doesn’t recognize that family, it rejects the node for trunking, and we end up with pods stuck Pending.

The failure mode is:

Karpenter provisions r8a.xlarge
control plane VPC Resource Controller rejects trunking for that instance type
node never advertises usable vpc.amazonaws.com/pod-eni
scheduler leaves pods Pending with Insufficient vpc.amazonaws.com/pod-eni
We see the controller rejection directly in node events:

Unsupported: The instance type r8a.xlarge is not supported yet by the vpc resource controller

On “upgrading the VPC resource controller”
Was there anything preventing an upgrade to a newer VPC resource controller version?

This is where we’re blocked / confused operationally:

The VPC Resource Controller appears to be EKS control-plane managed, not something we deploy/upgrade ourselves.
We don’t see a knob to pin/upgrade it independently, and the only visibility we currently have is via node events like:
ControllerVersionNotice: The node is managed by VPC resource controller version v1.7.15

So from an operator standpoint, we can’t reliably “upgrade VPC RC before upgrading Karpenter”, because:

we don’t control the rollout timing of the control plane component, and
we don’t have a supported mechanism to ensure it is upgraded (or even to check it proactively) other than observing events after nodes come up.
If there is a supported AWS/EKS mechanism to trigger/confirm the VPC Resource Controller version upgrade (or a reliable API/CLI path to query it), could you point us to it? Right now the only place we’ve been able to see the version is through the ControllerVersionNotice event.

Ask / what would help
Given this is fundamentally a version skew between two independently moving compatibility lists, the best mitigation we’ve found is denylisting instance families in NodePools (like excluding r8a). But that’s reactive and requires us to keep up with new family releases.

Would you consider one of the following (even as an optional feature/config)?

• A way to make Karpenter conservative for vpc.amazonaws.com/pod-eni by only allowing instance types that are known-compatible with the current cluster’s VPC RC version (if detectable), or
• a documented/supported “operational pattern” to avoid this class of outage (e.g., an official allowlist source to use, or guidance on how to keep Karpenter + VPC RC compatibility aligned on EKS).

[ryan-mist]

I see, thanks for the clarification!

It looks like the VPC resource controller version is tied to the EKS Platform version - https://github.com/aws/amazon-vpc-resource-controller-k8s/releases

If there is a supported AWS/EKS mechanism to trigger/confirm the VPC Resource Controller version upgrade (or a reliable API/CLI path to query it), could you point us to it?

When a new platform version is release, your cluster should be upgraded automatically (but it may take a little time) - https://docs.aws.amazon.com/eks/latest/userguide/platform-versions.html?icmpid=docs_eks_help_panel_hp_cluster_k8s_version

You can check the VPC Resource controller version by correlating your platform version with controllers release notes

aws eks describe-cluster --name {CLUSTER_NAME} --query cluster.platformVersion

---

For an existing operational pattern, I'd imagine it would be to change your NodePool requirements so it can only launch instance types that are supported by the version of the VPC Resource Controller your cluster is using. I know that this is a bit of a configuration burden though.

I'll talk with the team to see if we've discussed anything related to this in the past / to get an better path forward.

[kamal-kailasababu]

I see, thanks for the clarification!

It looks like the VPC resource controller version is tied to the EKS Platform version - https://github.com/aws/amazon-vpc-resource-controller-k8s/releases

If there is a supported AWS/EKS mechanism to trigger/confirm the VPC Resource Controller version upgrade (or a reliable API/CLI path to query it), could you point us to it?

When a new platform version is release, your cluster should be upgraded automatically (but it may take a little time) - https://docs.aws.amazon.com/eks/latest/userguide/platform-versions.html?icmpid=docs_eks_help_panel_hp_cluster_k8s_version

You can check the VPC Resource controller version by correlating your platform version with controllers release notes

aws eks describe-cluster --name {CLUSTER_NAME} --query cluster.platformVersion

For an existing operational pattern, I'd imagine it would be to change your NodePool requirements so it can only launch instance types that are supported by the version of the VPC Resource Controller your cluster is using. I know that this is a bit of a configuration burden though.

I'll talk with the team to see if we've discussed anything related to this in the past / to get an better path forward.

@ryan-mist ,
Thanks, but I think the key point is being missed: this isn’t just a “configuration burden”, it’s an unbounded operational risk caused by a control-plane/data-plane race we don’t control.

Core issue (risk, not misconfig)

Even if we configure NodePools “correctly” today, this can still recur whenever:

1. AWS releases a new EC2 family and Karpenter includes it in its generated limits list, and
2. the EKS control-plane VPC Resource Controller in our cluster/region is still on an older platform version for some period.

During that window, Karpenter can provision nodes that the control-plane VPC RC rejects for trunking, and SG-for-Pods workloads can hard-stall (pods Pending with Insufficient vpc.amazonaws.com/pod-eni). That’s a platform availability risk for us, not just an inconvenience.

On “EKS upgrades automatically”

I understand the platform version should eventually roll forward automatically, but that’s exactly the problem: we cannot control when it happens, and the “may take a little time” window is precisely when we can be impacted.

So “upgrade your platform version” isn’t an actionable mitigation for us, because:

• we can’t force/trigger that rollout, and
• we only learn we’re behind once the issue happens (or by manually correlating platform version ↔ controller release notes).

What we need (actionable path)

Right now the only mitigation is manual allow/deny lists in NodePools, which is reactive and requires us to track EC2 family launches.

Concretely, we need one of:

1. A Karpenter-side guardrail for Pod-ENI workloads to avoid launching instance types that might not be supported by the cluster’s current VPC RC/platform version (even if that means being conservative), or
2. an officially supported operational pattern/automation (ideally documented) that:
   • queries the cluster’s platform version (aws eks describe-cluster --query cluster.platformVersion),
   • maps it to the VPC RC supported instance list,
   • and automatically updates NodePool constraints (so humans don’t have to track new families).

If Karpenter maintainers don’t want to maintain per-controller-version config, could Karpenter at least provide an extension point / config input where users supply the allowed trunking-compatible list (derived from their platform version), so Karpenter can enforce it when vpc.amazonaws.com/pod-eni is requested?

Clarifying question

Is there any plan or precedent for Karpenter to treat Pod-ENI as “control-plane compatibility sensitive” and bias towards safety (e.g., “unknown families default to not trunking-compatible”) to prevent this class of outage?

Happy to share the exact node/pod events and the NodePool spec we used, but the fundamental issue is the same: Karpenter can move faster than the EKS control plane compatibility list, and that can break SG-for-Pods scheduling.

[ryan-mist]

Discussed with the team, we will want to support this. Karpenter can internally maintain mappings of EKS platform versions (which we can discovered from DescribeCluster) to the VPC Resource Controller Versions as well as the supported instances types.