OCI Registry login failure since 3.0.0

[DrTrevorPhD]

Terraform, Provider, Kubernetes and Helm Versions

Terraform version: Terraform v1.9.5
Provider version: helm v3.0.2
Kubernetes version: Provider kubernetes v2.37.1/ cluster is on 1.31.9

Affected Resource(s)

• helm_release

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.
# Install cert-manager
# --------------------

###Resource
resource "helm_release" "cert_manager" {
  name                = "cert-manager"
  repository          = "oci://ouracr.azurecr.io/helm/oem/safe"
  chart               = "cert-manager"
  version             = "v1.15.3"
  repository_username = "00000000-0000-0000-0000-000000000000"
  repository_password = data.external.acr_token.result.accessToken
  pass_credentials    = false
  namespace           = var.system_namespace

  set = [{
    name  = "image.tag"
    value = "v1.15.3"
    },

    {
      name  = "image.repository"
      value = "ouracr.azurecr.io/oem/clean/cert-manager-controller"
    },

    # This provides the ClusterIssuer and Certifikate kinds used in the k8s YAML
    {
      name  = "crds.enabled"
      value = "true"
    },

    # Before cert-manager passes the TXT DNS challenge off to letsencrypt, it first checks to ensure the TXT record exists.
    # This is a problem because the default DNS is the Azure internal DNS (this is the assumed problem rather than a known
    # problem)... because the TXT record *must* exist on the public entry for letsencrypt to see it, we need to override
    # the default DNS to a public DNS service.
    {
      name  = "extraArgs"
      value = "{--dns01-recursive-nameservers-only,--dns01-recursive-nameservers=8.8.8.8:53\\,1.1.1.1:53}"
  }]

  depends_on = [azurerm_role_assignment.dns_zone_contributor_aks_mi]
  lifecycle {
    ignore_changes = [repository_password]
  }
}

# Put in a 1 min sleep because otherwise cert-manager isn't ready in time for certificate creation
# ------------------------------------------------------------------------------------------------
resource "time_sleep" "cert_manager_postcreate" {
  create_duration = "1m"
  depends_on      = [helm_release.cert_manager]
}

###Provider file excerpt 
provider "helm" {
  kubernetes = {
    host                   = data.azurerm_kubernetes_cluster.aks.kube_config.0.host
    username               = data.azurerm_kubernetes_cluster.aks.kube_config.0.username
    password               = data.azurerm_kubernetes_cluster.aks.kube_config.0.password
    client_key             = base64decode(data.azurerm_kubernetes_cluster.aks.kube_admin_config.0.client_key)
    client_certificate     = base64decode(data.azurerm_kubernetes_cluster.aks.kube_admin_config.0.client_certificate)
    cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.aks.kube_admin_config.0.cluster_ca_certificate)
  }
  registries = [
    {
      url      = "oci://ouracr.azurecr.io"
      username = "00000000-0000-0000-0000-000000000000"
      password = data.external.acr_token.result.accessToken
    }
  ]
}

Debug Output

NOTE: In addition to Terraform debugging, please set HELM_DEBUG=1 to enable debugging info from helm.
You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.
╷
│ Error: OCI Registry Login Failed
│
│ with helm_release.cert_manager,
│ on helm.cert-manager.tf line 14, in resource "helm_release" "cert_manager":
│ 14: resource "helm_release" "cert_manager" {
│
│ Failed to log in to OCI registry "oci://ouracr.azurecr.io/helm/oem/safe": could not login to OCI registry "ouracr.azurecr.io": Get "https://ouracr.azurecr.io/v2/": unauthorized:
│ {"errors":[{"code":"UNAUTHORIZED","message":"authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 2ad5aab6-fb80-4ce4-8ec9-39f52e108e78"}]}
╵
╷
│ Error: OCI Registry Login Failed
│
│ with helm_release.sealed-secrets,
│ on helm.sealedsecrets.tf line 1, in resource "helm_release" "sealed-secrets":
│ 1: resource "helm_release" "sealed-secrets" {
│
│ Failed to log in to OCI registry "oci://ouracr.azurecr.io/helm/oem/safe": could not login to OCI registry "ouracr.azurecr.io": Get "https://ouracr.azurecr.io/v2/": unauthorized:
│ {"errors":[{"code":"UNAUTHORIZED","message":"authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 9a6f2a50-f990-454c-9755-1e40f1ce25a2"}]}

Panic Output

Steps to Reproduce

1. terraform apply --var-file=ourvars.common.tfvars

Expected Behavior

Plan with 2 resources to change

Actual Behavior

│ Error: OCI Registry Login Failed
│
│ with helm_release.cert_manager,
│ on helm.cert-manager.tf line 14, in resource "helm_release" "cert_manager":
│ 14: resource "helm_release" "cert_manager" {
│
│ Failed to log in to OCI registry "oci://ouracr.azurecr.io/helm/oem/safe": could not login to OCI registry "ouracr.azurecr.io": Get "https://ouracr.azurecr.io/v2/": unauthorized:
│ {"errors":[{"code":"UNAUTHORIZED","message":"authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 2ad5aab6-fb80-4ce4-8ec9-39f52e108e78"}]}
╵
╷
│ Error: OCI Registry Login Failed
│
│ with helm_release.sealed-secrets,
│ on helm.sealedsecrets.tf line 1, in resource "helm_release" "sealed-secrets":
│ 1: resource "helm_release" "sealed-secrets" {
│
│ Failed to log in to OCI registry "oci://ouracr.azurecr.io/helm/oem/safe": could not login to OCI registry "ouracr.azurecr.io": Get "https://ouracr.azurecr.io/v2/": unauthorized:
│ {"errors":[{"code":"UNAUTHORIZED","message":"authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 9a6f2a50-f990-454c-9755-1e40f1ce25a2"}]}

Important Factoids

Important to note that if I comment out the registries block, it fails at plan. If I uncomment it, and I comment out registry_username and registry_password, it plans as expected but when I apply it throws the exact same error as seen above.

I have tried the Get "https://ouracr.azurecr.io/v2/" via postman using the exact token we have and I see a clean response. When I try without the token I get the exact error thrown above, please see below screenshots
With

Without

References

• 🌱 Bump k8s.io/client-go from 0.27.1 to 0.28.1 #1234

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[chriskuchin]

I am seeing this with ecr on aws using the ecrpublic data source for the username and password as well

resource "helm_release" "karpenter_crd" {
  namespace           = "kube-system"
  name                = "karpenter-crd"
  repository          = "oci://public.ecr.aws/karpenter"
  repository_username = data.aws_ecrpublic_authorization_token.token.user_name
  repository_password = data.aws_ecrpublic_authorization_token.token.password
  chart               = "karpenter-crd"
  version             = "1.5.0"
  wait                = false

[basigabri]

Getting AWS OCI 403 forbidden error also, with helm v3 provider

resource "helm_release" "internal-lb" {
  count               = var.create_internal_lb ? 1 : 0
  chart               = "my-custom-chart"
  version             = "1.0.0"
  name                = "my-custom-chart"
  namespace           = "ingress"
  repository          = "oci://${data.aws_caller_identity.current.account_id}.dkr.ecr.${var.region}.amazonaws.com"
  repository_username = data.aws_ecr_authorization_token.token.user_name
  repository_password = data.aws_ecr_authorization_token.token.password
  values = [
    file(var.helm_values)
  ]

}

terraform {
  required_providers {
    
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "2.36.0"
    }
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 3.0"
    }
  }
  required_version = "~> 1.9"
}

provider "aws" {
  region = var.region
}

provider "helm" {
  kubernetes = {
    host                   = data.terraform_remote_state.aws_eks_cluster.outputs.cluster_endpoint
    cluster_ca_certificate = base64decode(data.terraform_remote_state.aws_eks_cluster.outputs.cluster_certificate_authority_data)
    exec = {
      api_version = "client.authentication.k8s.io/v1beta1"
      args        = ["eks", "--region", var.region, "get-token", "--cluster-name", data.terraform_remote_state.aws_eks_cluster.outputs.cluster_name]
      command     = "aws"
    }
  }
  registries = [
    {
      name     = "default"
      url      = data.aws_ecr_authorization_token.token.proxy_endpoint
      username = data.aws_ecr_authorization_token.token.user_name
      password = data.aws_ecr_authorization_token.token.password
    }
  ]
}

Errors:

OCI registry "xxxx.dkr.ecr.eu-central-1.amazonaws.com": login
  │ attempt to https://xxxx.dkr.ecr.eu-central-1.amazonaws.com/v2/
  │ failed with status: 403 Forbidden

[jocel1]

Same issue here with karpenter:

│ Error: OCI Registry Login Failed
│ 
│   with helm_release.karpenter,
│   on main.tf line 325, in resource "helm_release" "karpenter":
│  325: resource "helm_release" "karpenter" {
│ 
│ Failed to log in to OCI registry "oci://public.ecr.aws/karpenter": could
│ not login to OCI registry "public.ecr.aws": login attempt to
│ https://public.ecr.aws/v2/ failed with status: 403 Forbidden
╵

terraform {
  required_version = ">= 1.11.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.34"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.10"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 3.0"
    }
    null = {
      source  = "hashicorp/null"
      version = ">= 3.0"
    }
    flux = {
      source  = "fluxcd/flux"
      version = "1.6.3"
    }
    vault = {
      source  = "hashicorp/vault"
      version = "5.0.0"
    }
  }
}

data "aws_ecrpublic_authorization_token" "token" {
 provider = aws.virginia
}

resource "helm_release" "karpenter_crd" {
 namespace        = "karpenter"
 create_namespace = true

 name                = "karpenter-crd"
 repository          = local.karpenter_repository
 repository_username = data.aws_ecrpublic_authorization_token.token.user_name
 repository_password = data.aws_ecrpublic_authorization_token.token.password
 chart               = "karpenter-crd"
 version             = local.karpenter_version
 wait                = true
 values              = []
}

resource "helm_release" "karpenter" {
 namespace        = "karpenter"
 create_namespace = true

 name                = "karpenter"
 repository          = local.karpenter_repository
 repository_username = data.aws_ecrpublic_authorization_token.token.user_name
 repository_password = data.aws_ecrpublic_authorization_token.token.password
 chart               = "karpenter"
 version             = local.karpenter_version
 wait                = false
 skip_crds           = true
}

[jocel1]

Note I also tried to completely remove the repository_username and repository_password + delete the associated .config/helm/registry/config.json, but it doesn't help

[edersonbrilhante-sbg-cisco]

@jocel1 I have upgraded the provider today and had the same issue.

Luckily I found this workaround and now it is working.

#1645 (comment)

[chriskuchin]

The state rm and state import workaround worked for me as well

[jocel1]

I also succeeded by tainting & recreating karpenter but a few days later the problem is back, so it's only a temp workaround unfortunately

[Invincibear]

Broken for me too, trying to pull oci://ghcr.io/actions/actions-runner-controller-charts using a PAT that only has access to public repos. Funny enough this is a copy/paste from a working deploying for another repo, pulling from the same repository using the same PAT. I even tried copy/pasting the state of the working one, no dice.

resource "helm_release" "gha_runner_netcore_repo" {
  count = terraform.workspace == "prod" ? 1 : 0
  name  = "gha-runners-netcore-repo"

  chart           = "gha-runner-scale-set"
  cleanup_on_fail = true
  repository      = "oci://ghcr.io/actions/actions-runner-controller-charts"
  wait            = true
  wait_for_jobs   = true

  create_namespace = true
  namespace        = "github-runners"

  # https://github.com/actions/actions-runner-controller/pkgs/container/actions-runner-controller-charts%2Fgha-runner-scale-set
  version = "0.10.1"

  # Required to access the public container repository
  # Create a PAT with default public repos permissions
  repository_password = data.azurerm_key_vault_secret.GH_READ_PUBLIC_REPOS.value

  set_sensitive {
    name  = "githubConfigSecret.github_app_private_key"
    value = base64decode(data.azurerm_key_vault_secret.GH_RUNNER_PRIVATE_KEY.value) # base64 -w0 filename.pem
  }

  # https://github.com/actions/actions-runner-controller/blob/master/charts/gha-runner-scale-set/values.yaml
  values = [data.github_repository_file.gha_runner_scale_set-netcore_repo-chart_values.content]
}

[Invincibear]

update: I had to add repository_username = 'owner of the PAT', now it works for me

[chriskuchin]

It appears that the credentials are getting stored in the state so state rm and import clears the creds and allows new ones to be set but once those expire the problem comes back. We are testing removing the credentials from the resource doing the rm/import and then using provider set credentials to see if it stops storing the expiring creds in the state.

I am not sure if this was an intentional change in behavior or forced by the sdk migration but will report back if the provider based creds with rm/import of the resource will solve it for us at least.

[chriskuchin]

That appears to have solved the problem for us. We removed the credentials from the resource and placed them on the provider, and then did a state rm/import to make sure the cached credentials were null and 24h post apply things are still working.

The bug/change in behavior appears to be that once applied resource level credentials are reused and with temporary creds this results in them expiring eventually and the system isn't reading the new fresh creds to update the state so it simply fails after reading the expired creds in the state

[kevinfrommelt]

I believe this is the line that is pulling the credentials from state rather than the current configuration. https://github.com/hashicorp/terraform-provider-helm/blob/main/helm/resource_helm_release.go#L765

ociDiags := OCIRegistryLogin(ctx, meta, actionConfig, meta.RegistryClient, state.Repository.ValueString(), state.Chart.ValueString(), state.RepositoryUsername.ValueString(), state.RepositoryPassword.ValueString())

[skibz]

It's disappointing to see no response from contributors, here, given the severity of this issue.

The Helm provider is literally hosing the Terraform state of deployed systems.

Fixing this needs to be prioritised.