helm_release does not wait for daemonsets to be ready

[c4milo]

Terraform, Provider, Kubernetes and Helm Versions

❯ terraform version
Terraform v1.7.5
on darwin_arm64
+ provider registry.terraform.io/hashicorp/azurerm v3.102.0
+ provider registry.terraform.io/hashicorp/helm v2.13.2
+ provider registry.terraform.io/hashicorp/kubernetes v2.29.0
+ provider registry.terraform.io/hashicorp/random v3.6.1

Your version of Terraform is out of date! The latest version
is 1.8.3. You can update by downloading from https://www.terraform.io/downloads.html

Affected Resource(s)

• helm_release

Terraform Configuration Files

locals {
  redpanda_node_setup = "redpanda-node-setup"
}

resource "kubernetes_namespace" "redpanda_node_setup" {
  metadata {
    name = local.redpanda_node_setup
  }
}

resource "helm_release" "redpanda_node_setup" {
  name              = local.redpanda_node_setup
  chart             = "${path.module}/helm_charts/node-setup"
  version           = var.redpanda_node_setup_chart_version
  namespace         = kubernetes_namespace.redpanda_node_setup.id
  atomic            = true
  wait              = true
  wait_for_jobs     = true
  skip_crds         = true
  lint              = true
  cleanup_on_fail   = true  # it should be safe as long as we skip installing CRDs through Helm.
  verify            = false # enable cryptographic verification of provenance
  dependency_update = false
  create_namespace  = false
  max_history       = 5
  timeout           = 300 # 5min, destroy operations take a while.

  values = [
    "${templatefile("${path.module}/templates/redpanda_node_setup.helm.values.yml.tftpl", {
      node_setup_script = file("${path.module}/files/node_setup.bash")
    })}"
  ]
}

Debug Output

module.cert_manager.kubernetes_manifest.cert_manager_crds["files/crds/cert_manager.crd.2.yml"]: Creating...
2024-05-09T18:28:47.502-0400 [INFO]  Starting apply for module.cert_manager.kubernetes_manifest.cert_manager_crds["files/crds/cert_manager.crd.2.yml"]
2024-05-09T18:28:47.502-0400 [DEBUG] skipping FixUpBlockAttrs
2024-05-09T18:28:47.506-0400 [DEBUG] module.cert_manager.kubernetes_manifest.cert_manager_crds["files/crds/cert_manager.crd.2.yml"]: applying the planned Create change
2024-05-09T18:28:47.966-0400 [INFO]  provider.terraform-provider-helm_v2.13.2_x5: 2024/05/09 18:28:47 [DEBUG] creating 2 resource(s): timestamp=2024-05-09T18:28:47.966-0400
2024-05-09T18:28:48.025-0400 [INFO]  provider.terraform-provider-helm_v2.13.2_x5: 2024/05/09 18:28:48 [DEBUG] beginning wait for 2 resources with timeout of 5m0s: timestamp=2024-05-09T18:28:48.025-0400
module.redpanda_node_setup.helm_release.redpanda_node_setup: Creation complete after 1s [id=redpanda-node-setup]
2024-05-09T18:28:48.115-0400 [INFO]  provider.terraform-provider-helm_v2.13.2_x5: 2024/05/09 18:28:48 [DEBUG] creating 8 resource(s): timestamp=2024-05-09T18:28:48.115-0400
2024-05-09T18:28:48.229-0400 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot

Panic Output

Steps to Reproduce

1. terraform apply

Expected Behavior

I was expecting it will wait for 3 daemonset pods to be ready before continuing.

Actual Behavior

It continued to the next resource right after scheduling the daemonset

Important Factoids

References

• 🌱 Bump k8s.io/client-go from 0.27.1 to 0.28.1 #1234

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[iamunni]

@jrhouston I'd like to take a look at this

[github-actions]

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

[aa2811]

Please don't close this! We've recently hit this while installing the csi-secrets-store-provider-azure chart, which includes a DaemonSet here: https://github.com/Azure/secrets-store-csi-driver-provider-azure/blob/master/charts/csi-secrets-store-provider-azure/templates/provider-azure-installer.yaml#L3

We'd got the image path wrong for the pod, but weren't alerted to this fact until we logged onto the cluster and checked the pod health, where pods were failing with ErrImagePull.

Particularly important because Microsoft recently changed the default URL for this image: Diff

We use an intermediate container caching solution, so this broke our install, because we didn't catch the updated URL. If the helm provider was catching unready DaemonSets, we would have caught this earlier.