kube_client::client: Unsuccessful data error parse: 404 page not found warnings on EKS

[devfacet]

First of all, thank you for this great tool.

I deployed the operator to an EKS cluster and everything works fine, but it periodically produces the following warning:

restate-operator 2025-06-24T17:02:54.930787Z  WARN kube_client::client: Unsuccessful data error parse: 404 page not found

What's the best way debug this warning?

[jackkleeman]

Hi @devfacet - are both CRDs installed ie RestateDeployment and RestateCluster? I have seen this error before when I didn't have the RestateDeployment crd installed. I can fix the operator to properly exit if this is the case, perhaps

[devfacet]

No, I only installed RestateCluster CRD. If that's the case then I can install it.

[jackkleeman]

Yeah, the 404 is caused by the Deployment not being installed. Its harmless for now but might be a fatal error in future

[devfacet]

@jackkleeman: I can confirm that the issue is solved after applying the RestateDeployment CRD.

For those who use Terraform to manage this operator on EKS, this is what I do:

1. Generate the K8s resource file:

helm template restate-operator oci://ghcr.io/restatedev/restate-operator-helm \
  --namespace restate-operator \
  --set version=1.7.2 \
  --set serviceAccount.create=true \
  > restate-operator.yaml

2. Prepend restateclusters.yaml and restatedeployments.yaml to restate-operator.yaml

3. Use terraform-provider-kubectl to apply:

...
...

resource "kubernetes_namespace" "restate_operator" {
  count = var.restate_operator_enabled ? 1 : 0

  metadata {
    name = local.restate_operator_namespace
    labels = merge(
      local.cluster_default_labels,
      {
        "app.kubernetes.io/name"       = "${local.restate_operator_namespace}",
        "app.kubernetes.io/managed-by" = var.aws_tag_managedby
      }
    )
    annotations = local.cluster_default_annotations
  }
}

data "kubectl_file_documents" "restate_operator" {
  count = var.restate_operator_enabled ? 1 : 0

  content = templatefile(
    "${path.module}/templates/restate-operator-v${var.restate_operator_version}.yaml.tftpl",
    {
      "OPERATOR_NAMESPACE" = "${local.restate_operator_namespace}",
    }
  )
}

resource "kubectl_manifest" "restate_operator" {
  for_each  = local.restate_operator_manifests
  yaml_body = each.value

  depends_on = [
    kubernetes_namespace.restate_operator,
    data.kubectl_file_documents.restate_operator,
  ]
}