Skip to content

create-acm-cert selects a private hosted zone for public ACM DNS validation in split-horizon Route 53; certificate stays in PENDING_VALIDATION #4840

Description

@niv1612

Bug Description

Thanks for the certificate-management feature - auto-provisioning ACM certs from an Ingress is really handy. We ran into an edge case with split-horizon Route 53 and wanted to share it with a couple of possible directions.

When create-acm-cert provisions an Amazon-issued (public) ACM certificate, the controller resolves the DNS validation record into the most-specific matching Route 53 hosted zone. In a split-horizon setup (a private zone that is a subdomain of a public zone), that most-specific zone is the private one. Public ACM validation is checked over public DNS, so a record placed in a private-only zone is never resolvable, and the certificate remains in PENDING_VALIDATION - the HTTPS listener never finalizes and the Ingress gets no address.

Steps to Reproduce

  • Route 53 in one account with a private hosted zone internal.example.com and a public hosted zone example.com.
  • Enable the EnableCertificateManagement feature gate; create an Ingress with host app.internal.example.com and annotation alb.ingress.kubernetes.io/create-acm-cert: "true" (no certificate-arn).
  • Controller logs show it building the model, requesting the certificate, and creating the validation record — the log lines include only the record name/value, not a zone:
    "msg":"created validation record","record":{"Name":".app.internal.example.com.","Type":"CNAME","Value":".xxxxxx.acm-validations.aws."}
  • The validation CNAME ends up in the private internal.example.com zone.
  • The ACM certificate stays PENDING_VALIDATION and the Ingress never receives an address.

Expected Behavior

For a public (Amazon-issued) certificate, the validation record should be created in a hosted zone where public ACM validation can actually succeed - a public zone - or the controller should allow the user to specify which zone to use.

Actual Behavior

The validation record is created in the private zone, where public ACM validation can never complete, so the certificate stays PENDING_VALIDATION indefinitely and the affected Ingress never serves HTTPS. It does not stop the controller overall. Reproducible Always in a split-horizon topology.

Regression
Was the functionality working correctly in a previous version ? [Yes / No]
If yes, specify the last version where it worked as expected
No - this appears to be the behavior since the certificate-management feature was introduced (PR #4554). Not a regression from a previously working version.

Current Workarounds

Pre-issue the certificate (DNS-validated by placing the CNAME in the public zone) and attach it via the certificate-arn annotation, which bypasses the controller's Route 53 logic entirely.

Environment

  • AWS Load Balancer controller version: v3.3.0

  • Kubernetes version: 1.35 (EKS)

  • Using EKS (yes/no), if so version?: Yes, 1.35

  • Using Service or Ingress: Ingress

  • AWS region: eu-central-1

  • How was the aws-load-balancer-controller installed:
    the upstream chart is rendered to static manifests (Tanka helm template) and applied via ArgoCD (GitOps), so helm ls/helm get values don't apply. Equivalent render command (chart v3.3.0 from https://aws.github.io/eks-charts):

    helm template aws-load-balancer-controller eks/aws-load-balancer-controller \
      --version 3.3.0 \
      --namespace alb \
      --set clusterName=<cluster-name> \
      --set region=eu-central-1 \
      --set vpcId=<vpc-id> \
      --set enableCertManager=true \
      --set controllerConfig.featureGates.EnableCertificateManagement=true \
      --set serviceAccount.create=true \
      --set serviceAccount.name=aws-load-balancer-controller \
      --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=<controller-iam-role-arn>
      Controller image: public.ecr.aws/eks/aws-load-balancer-controller:v3.3.0.
  • Current state of the Controller configuration:

    kubectl -n describe deployment aws-load-balancer-controller
    Name:                   aws-load-balancer-controller
    Namespace:              alb
    CreationTimestamp:      Wed, 20 May 2026 14:54:03 
    Labels:                 app.kubernetes.io/instance=aws-load-balancer-controller
                            app.kubernetes.io/managed-by=x
                            app.kubernetes.io/name=aws-load-balancer-controller
                            app.kubernetes.io/version=v3.3.0
                            helm.sh/chart=aws-load-balancer-controller-3.3.0
    Annotations:            argocd.argoproj.io/tracking-id: x:apps/Deployment:alb/aws-load-balancer-controller
                            deployment.kubernetes.io/revision: 5
    Selector:               app.kubernetes.io/instance=aws-load-balancer-controller,app.kubernetes.io/name=aws-load-balancer-controller
    Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
    StrategyType:           RollingUpdate
    MinReadySeconds:        0
    RollingUpdateStrategy:  25% max unavailable, 25% max surge
    Pod Template:
      Labels:           app.kubernetes.io/instance=aws-load-balancer-controller
                        app.kubernetes.io/name=aws-load-balancer-controller
      Annotations:      automation.x/restartedAt: 2026-07-05T07:07:27Z
                        kubectl.kubernetes.io/restartedAt: 2026-05-24T15:05:12+03:00
                        prometheus.io/port: x
                        prometheus.io/scrape: true
      Service Account:  aws-load-balancer-controller
      Containers:
       aws-load-balancer-controller:
        Image:       public.ecr.aws/eks/aws-load-balancer-controller:v3.3.0
        Ports:       x/TCP (webhook-server), x/TCP (metrics-server)
        Host Ports:  x/TCP (webhook-server), x/TCP (metrics-server)
        Args:
          --cluster-name=x
          --ingress-class=x
          --aws-region=eu-central-1
          --aws-vpc-id=x
          --feature-gates=ALBGatewayAPI=true,EnableCertificateManagement=true
        Liveness:     http-get http://:x/x delay=Xs timeout=Xs period=Xs #success=x #failure=x
        Readiness:    http-get http://:x/readyz delay=Xs timeout=Xs period=Xs #success=x #failure=x
        Environment:  <none>
        Mounts:
          /x/x/x from cert (ro)
      Volumes:
       cert:
        Type:               Secret (a volume populated by a Secret)
        SecretName:         x
        Optional:           false
      Priority Class Name:  x
      Node-Selectors:       <none>
      Tolerations:          <none>
    Conditions:
      Type           Status  Reason
      ----           ------  ------
      Progressing    True    NewReplicaSetAvailable
      Available      True    MinimumReplicasAvailable
    OldReplicaSets:  aws-load-balancer-controller-69d7854d6c (0/0 replicas created), aws-load-balancer-controller-d7f6989b6 (0/0 replicas created), aws-load-balancer-controller-5c6ffd6844 (0/0 replicas created), aws-load-balancer-controller-856d9dc7dc (0/0 replicas created)
    NewReplicaSet:   aws-load-balancer-controller-fcb5cfb5 (2/2 replicas created)
    Events:          <none>

Possible Solution (Optional)

From reading the code, GetHostedZoneID (pkg/aws/services/route53.go) matches the hosted zone by DNS suffix and doesn't currently consult zone.Config.PrivateZone; listHostedZones returns both public and private zones. PR #4785 ("match by longest suffix", v3.4.1) keeps the most-specific-zone behavior, so it doesn't change this case (apologies if I missed an existing option). Two directions that would close the gap for public certificates:

  1. Explicit hosted-zone override — let the user specify the Route 53 hosted zone ID used for ACM validation (e.g. an Ingress annotation alb.ingress.kubernetes.io/acm-validation-hosted-zone-id, or an IngressClassParams field). Deterministic; covers split-horizon, delegated subdomains, and cross-account DNS where suffix inference alone isn't enough.
  2. Public-zone-aware automatic selection — for Amazon-issued certificates, restrict candidate zones to public hosted zones and walk up to the nearest public ancestor. This keeps the existing "most specific matching zone" behavior, narrowed to zones that can satisfy public validation; it handles the common split-horizon case with no per-Ingress config.

Either resolves the reported case; both together (auto default + override) would be ideal. It would also help to fail fast with a clear error when no public zone matches and no override is given, rather than create a certificate that can't be validated and leave it stuck.

Happy to help test or contribute a PR if a direction sounds reasonable.
Contribution Intention (Optional)

  • Yes, I'm willing to submit a PR to fix this issue
  • No, I cannot work on a PR at this time

Additional Context

Reference: AWS - Issue an ACM certificate for a domain in a private hosted zone (public-cert validation records must live in a public zone). This affects anyone with split-horizon DNS or overlapping private/public zones.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions