First helm release always succeeds and doesn't wait for all pods running

[FischlerA]

Terraform, Provider, Kubernetes and Helm Versions

Terraform version: 0.14.4
Provider version: 2.0.2
Kubernetes version: AWS EKS 1.18
Helm version: 3

Affected Resource(s)

• helm_release

Debug Output

https://gist.github.com/FischlerA/7930aff18d68a7b133ff22aadc021517

Steps to Reproduce

1. terraform apply

Expected Behavior

The helm deployment should fail since the pod that is being deployed is running an image that will always fail. (private image which i can't share)

Actual Behavior

The first time the helm release is deployed it always succeeds after reaching the timeout (5 min), any further deployments fail as they are supposed to after reaching the timeout (5min).

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jrhouston]

Thanks for opening @FischlerA. Did you try using the wait attribute? By default helm will not wait for all pods to become ready, just create the API resources.

[FischlerA]

Thanks for opening @FischlerA. Did you try using the wait attribute? By default helm will not wait for all pods to become ready, just create the API resources.

Per documentation the wait attributed defaults to true. But even after explicitly setting it to true the behavior didn't change and it was still seen as success with a crashing pod.

[jrhouston]

Ah yep, you're right – I will try and reproduce this.

The provider itself doesn't do the waiting, it just passes along the wait flag to the install action in the helm package. Do you get the same issue if you do a helm install --wait with your chart?

[isabellemartin2]

@jrhouston
We deployed the chart again by using helm directly with helm install --wait and the behaviour was as expected:
After waiting for five minutes, we've got an error-message Error: timed out waiting for the condition.

[dinandr]

I had same experiences when I use helm_release in terraform, if something goes wrong, pod status is stay at "pending" or "Error", "CreateContainer" or some other unusual status for a little longer time, Helm terraform provider will not wait until pods are running, it will exit and reported completed, However terraform state was update as failed.

[whiskeysierra]

Saw the same behavior today when I deployed ingress-nginx and the very first job failed because it was rejected by another webhook. The terraform apply run waited for 5 minutes but reported a success, even though not a single resource was created successful. In fact only 1 job was there it was rejected.

[FischlerA]

@jrhouston were you able to take a look at this?

[jgreat]

I'm running into this too. I pretty regularly have a successful terraform apply (everything shows successful and complete) and end up with helm_release resources that show ~ status = "failed" -> "deployed" on a second run.

[loreleimccollum-work]

I think we are hitting this as well but not entirely sure.. we are seeing helm_release pass on first run with (wait = true) where not all the pods come online because of a Gatekeeper/PSP we have in the cluster, we are not sure how get our helm_release to fail in that case

[thecb4]

Hi all. I'm new to Terraform. I've had to split up my Terraform deployments and include a time_sleep because of this issue. Looking forward to an update here.

[descampsk]

Same thing with helm job and wait_for_jobs = true.
It will wait the timeout and then will return true.
If i reapply I got the following :

$ terraform apply -var image_tag=dev-ed4854d
helm_release.job_helm_release: Refreshing state... [id=api-migration]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.job_helm_release.helm_release.job_helm_release will be updated in-place
  ~ resource "helm_release" "job_helm_release" {
        id                         = "api-migration"
        name                       = "api-migration"
      ~ status                     = "failed" -> "deployed"
        # (24 unchanged attributes hidden)


        # (22 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

[vinothkumarsubs2019]

I faced this issue, helm-release 'timeout' options seem not working, helm-relese stated as "successfully completed" with in 5 seconds , even though PODs are init stage.