CSI driver crashes immediately on Kubernetes 1.34 (controller pod)

[angustohrallegrinski]

---

Description

The Hetzner CSI driver (hcloud-csi-driver:v2.18.3) starts in the controller pod, creates the socket /run/csi/socket, but then exits immediately without producing any logs. The sidecar containers (csi-attacher, csi-provisioner, csi-resizer) report:

Lost connection to CSI driver, exiting
grpc: addrConn.createTransport failed to connect to {Addr: "/run/csi/socket"}.
error reading server preface: read unix @->/run/csi/socket: use of closed network connection

---

Environment

• Kubernetes version: v1.34 latest
• CSI driver version: v2.18.3
• Sidecar versions:
  • csi-attacher: v4.10.0
  • csi-provisioner: v6.1.0
  • csi-resizer: v2.0.0
  • liveness-probe: v2.17.0
• Manifest: Based on the official Hetzner manifest (includes RBAC, CSIDriver, StorageClass).
• Secret: Present (hcloud with key token), token is valid.
• Network: Pod has connectivity to https://api.hetzner.cloud.

---

Expected Behavior

The CSI driver should remain running and respond to sidecar requests.

---

Observed Behavior

• The hcloud-csi-driver container in the controller:
  • Starts.
  • Creates the socket /run/csi/socket.
  • Exits immediately (Exit Code: 2) without any logs.
• No RBAC errors in logs.
• NodePlugin (hcloud-csi-node) works correctly.

---

Steps to Reproduce

1. Deploy the official Hetzner CSI manifest on Kubernetes 1.34.
2. Create the Secret hcloud with a valid token.
3. Observe the hcloud-csi-controller pod:

   kubectl -n kube-system get pods
   kubectl -n kube-system logs hcloud-csi-controller-xxx -c hcloud-csi-driver

4. Verify that the container exits immediately without logs.

---

Pod Events

State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 2

---

Question

• Is this a known issue with Kubernetes 1.34?
• Is there any additional configuration required for the controller?
• How can we enable more detailed logging to diagnose the crash?

[lukasmetzner]

Hey, can you please generate full logs for the csi controller pod, by including all containers and previous instances:

kubectl -n kube-system logs hcloud-csi-controller-xxx --all-containers --prefix --timestamps --previous

Are their any Kubernetes events related to the csi-driver?

[angustohrallegrinski]

Hello, this is the log.

[pod/hcloud-csi-controller-694cd4d6-k88b7/csi-provisioner] 2026-01-21T08:54:32.773895127Z W0121 08:54:32.773737 1 feature_gate.go:352] Setting GA feature gate Topology=true. It will be removed in a future release.
[pod/hcloud-csi-controller-694cd4d6-k88b7/csi-provisioner] 2026-01-21T08:54:32.773920486Z I0121 08:54:32.773836 1 csi-provisioner.go:170] "Version" version="v6.1.0"
[pod/hcloud-csi-controller-694cd4d6-k88b7/csi-provisioner] 2026-01-21T08:54:32.773923235Z I0121 08:54:32.773844 1 csi-provisioner.go:193] Building kube configs for running in cluster...
[pod/hcloud-csi-controller-694cd4d6-k88b7/csi-provisioner] 2026-01-21T08:54:32.782311106Z W0121 08:54:32.782244 1 logging.go:55] [core] [Channel #1 SubChannel #2]grpc: addrConn.createTransport failed to connect to {Addr: "/run/csi/socket", ServerName: "localhost", Attributes: {"<%!p(networktype.keyType=grpc.internal.transport.networktype)>": "unix" }, }. Err: connection error: desc = "transport: Error while dialing: dial unix /run/csi/socket: connect: connection refused"
[pod/hcloud-csi-controller-694cd4d6-k88b7/csi-provisioner] 2026-01-21T08:54:34.632159246Z W0121 08:54:34.631935 1 logging.go:55] [core] [Channel #1 SubChannel #2]grpc: addrConn.createTransport failed to connect to {Addr: "/run/csi/socket", ServerName: "localhost", Attributes: {"<%!p(networktype.keyType=grpc.internal.transport.networktype)>": "unix" }, }. Err: connection error: desc = "error reading server preface: read unix @->/run/csi/socket: use of closed network connection"
[pod/hcloud-csi-controller-694cd4d6-k88b7/csi-provisioner] 2026-01-21T08:54:35.482558025Z E0121 08:54:35.482316 1 connection.go:208] "Lost connection" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-694cd4d6-k88b7/csi-provisioner] 2026-01-21T08:54:35.482900630Z E0121 08:54:35.482542 1 connection.go:116] "Lost connection to CSI driver, exiting"
Error from server (BadRequest): previous terminated container "liveness-probe" in pod "hcloud-csi-controller-694cd4d6-k88b7" not found

I think is something related to the container that opens the local socket. It does not start, no log from it, maybe an error from the entrypoint?

[angustohrallegrinski]

If the dockerfile is this: https://github.com/hetznercloud/csi-driver/blob/main/Dockerfile
I think that it will be better to launch /bin/hcloud-csi-driver from CMD and not ENTRYPOINT (used for cheks and configurations), maybe some logs will appair

[phoenix-pnp]

I can confirm the error for my dev cluster.
After downgrading K3S to 1.33.7 all is working fine again

[angustohrallegrinski]

Thanks @phoenix-pnp !!
But in the documentation it says that Kubernetes 1.34 is supported, and I'm using that, I don't want to downgrade.

[lukasmetzner]

@angustohrallegrinski Weird that there are no previous containers of the hcloud-csi-controller.

Error from server (BadRequest): previous terminated container "liveness-probe" in pod "hcloud-csi-controller-694cd4d6-k88b7" not found

Could you maybe kubectl describe the pod for me? Do you have any other indiciations in your cluster, why this container is not starting?

[angustohrallegrinski]

@phoenix-pnp I have the same problem also in 1.33.7
The cluster configuration is:

• hetzner
• only private networks (--apiserver-advertise-address=xx on init)
• coredns does not start until I deply hetzner ccm-networks

I don't know why it not start! No logs because the application is configured to start into the entrypoint (but it's better to lounch it with CMD into Dockerfile!! and this is why there are no logs).

This is the describe output:
root@msa-k8s2-controller-01:~# kubectl describe pod hcloud-csi-controller-fb7688b79-2wql7 -n kube-system
Name: hcloud-csi-controller-fb7688b79-2wql7
Namespace: kube-system
Priority: 0
Service Account: hcloud-csi-controller
Node: msa-k8s2-worker-02/10.98.0.6
Start Time: Wed, 21 Jan 2026 12:17:54 +0000
Labels: app=hcloud-csi-controller
app.kubernetes.io/component=controller
app.kubernetes.io/instance=hcloud-csi
app.kubernetes.io/name=hcloud-csi
pod-template-hash=fb7688b79
Annotations:
Status: Running
IP: 10.244.3.3
IPs:
IP: 10.244.3.3
Controlled By: ReplicaSet/hcloud-csi-controller-fb7688b79
Containers:
csi-attacher:
Container ID: containerd://f1f061f6a3cf4f4a639acd0e0cc0f7db0087e0d316e242c3f20a73f8b4a2e98e
Image: registry.k8s.io/sig-storage/csi-attacher:v4.10.0
Image ID: registry.k8s.io/sig-storage/csi-attacher@sha256:be59d0556508d3dd419cc34c53062f170a28902ef36e832b947c7796458a083d
Port:
Host Port:
Args:
--default-fstype=ext4
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 21 Jan 2026 12:21:15 +0000
Finished: Wed, 21 Jan 2026 12:21:45 +0000
Ready: False
Restart Count: 5
Environment:
Mounts:
/run/csi from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8dxkx (ro)
csi-resizer:
Container ID: containerd://26634a06427cc2935451519af644121b2f4f7d12def41bbc73444eef73dab5e7
Image: registry.k8s.io/sig-storage/csi-resizer:v2.0.0
Image ID: registry.k8s.io/sig-storage/csi-resizer@sha256:4a95d94e57ad82f6977cd8d4fdcfcfc0b83f02d990e4e7715b688c20970a906d
Port:
Host Port:
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 21 Jan 2026 12:21:15 +0000
Finished: Wed, 21 Jan 2026 12:21:45 +0000
Ready: False
Restart Count: 5
Environment:
Mounts:
/run/csi from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8dxkx (ro)
csi-provisioner:
Container ID: containerd://4380ece2aeb524db17480e9fa84a82ea53f3b9fd4938ed70642bcca6740231b1
Image: registry.k8s.io/sig-storage/csi-provisioner:v6.1.0
Image ID: registry.k8s.io/sig-storage/csi-provisioner@sha256:e5900dc98b0d02f317ea9572f3824983c47a2fb5d5fe2b160aedf1b99c14c9e4
Port:
Host Port:
Args:
--feature-gates=Topology=true
--default-fstype=ext4
--extra-create-metadata
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 21 Jan 2026 12:21:15 +0000
Finished: Wed, 21 Jan 2026 12:21:45 +0000
Ready: False
Restart Count: 5
Environment:
Mounts:
/run/csi from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8dxkx (ro)
liveness-probe:
Container ID: containerd://fbbf6073d9debbcf4844b6e41d94fca835589d402e54517d4f80029a6624b73f
Image: registry.k8s.io/sig-storage/livenessprobe:v2.17.0
Image ID: registry.k8s.io/sig-storage/livenessprobe@sha256:9b75b9ade162136291d5e8f13a1dfc3dec71ee61419b1bfc112e0796ff8a6aa9
Port:
Host Port:
State: Running
Started: Wed, 21 Jan 2026 12:18:07 +0000
Ready: True
Restart Count: 0
Environment:
Mounts:
/run/csi from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8dxkx (ro)
hcloud-csi-driver:
Container ID: containerd://d0bd7222b9842a0d456c6214791d9128bbf64d7b78a47e9d4abdda003fe46e9a
Image: docker.io/hetznercloud/hcloud-csi-driver:v2.18.3
Image ID: docker.io/hetznercloud/hcloud-csi-driver@sha256:53b4be0777cd6b1231e7d1c4a371ec2e110cc4a06b212befdced3d012ff6e29b
Ports: 9189/TCP, 9808/TCP
Host Ports: 0/TCP, 0/TCP
Args:
-controller
State: Running
Started: Wed, 21 Jan 2026 12:21:45 +0000
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Wed, 21 Jan 2026 12:20:06 +0000
Finished: Wed, 21 Jan 2026 12:20:24 +0000
Ready: True
Restart Count: 6
Liveness: http-get http://:healthz/healthz delay=10s timeout=3s period=2s #success=1 #failure=5
Environment:
CSI_ENDPOINT: unix:///run/csi/socket
METRICS_ENDPOINT: 0.0.0.0:9189
ENABLE_METRICS: true
KUBE_NODE_NAME: (v1:spec.nodeName)
HCLOUD_TOKEN: <set to the key 'token' in secret 'hcloud'> Optional: false
Mounts:
/run/csi from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8dxkx (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
socket-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
kube-api-access-8dxkx:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
Optional: false
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Scheduled 4m7s default-scheduler Successfully assigned kube-system/hcloud-csi-controller-fb7688b79-2wql7 to msa-k8s2-worker-02
Normal Pulling 4m6s kubelet Pulling image "registry.k8s.io/sig-storage/csi-attacher:v4.10.0"
Normal Pulled 4m3s kubelet Successfully pulled image "registry.k8s.io/sig-storage/csi-attacher:v4.10.0" in 1.841s (3.171s including waiting). Image size: 40140287 bytes.
Normal Pulling 4m3s kubelet Pulling image "registry.k8s.io/sig-storage/csi-resizer:v2.0.0"
Normal Pulled 4m kubelet Successfully pulled image "registry.k8s.io/sig-storage/csi-resizer:v2.0.0" in 2.378s (3.08s including waiting). Image size: 40255661 bytes.
Normal Pulling 4m kubelet Pulling image "registry.k8s.io/sig-storage/csi-provisioner:v6.1.0"
Normal Started 3m55s kubelet Started container csi-provisioner
Normal Created 3m55s kubelet Created container: csi-provisioner
Normal Created 3m55s kubelet Created container: liveness-probe
Normal Pulled 3m55s kubelet Container image "registry.k8s.io/sig-storage/livenessprobe:v2.17.0" already present on machine
Normal Pulled 3m55s kubelet Successfully pulled image "registry.k8s.io/sig-storage/csi-provisioner:v6.1.0" in 2.205s (5.096s including waiting). Image size: 42386610 bytes.
Normal Started 3m54s kubelet Started container liveness-probe
Normal Pulled 3m54s kubelet Container image "docker.io/hetznercloud/hcloud-csi-driver:v2.18.3" already present on machine
Normal Started 3m54s kubelet Started container hcloud-csi-driver
Normal Created 3m54s kubelet Created container: hcloud-csi-driver
Normal Created 3m51s (x2 over 4m) kubelet Created container: csi-resizer
Normal Created 3m51s (x2 over 4m3s) kubelet Created container: csi-attacher
Normal Started 3m51s (x2 over 4m) kubelet Started container csi-resizer
Normal Started 3m51s (x2 over 4m3s) kubelet Started container csi-attacher
Normal Pulled 3m51s kubelet Container image "registry.k8s.io/sig-storage/csi-attacher:v4.10.0" already present on machine
Normal Pulled 3m51s kubelet Container image "registry.k8s.io/sig-storage/csi-resizer:v2.0.0" already present on machine
Normal Pulled 3m51s kubelet Container image "registry.k8s.io/sig-storage/csi-provisioner:v6.1.0" already present on machine
Warning BackOff 3m16s (x5 over 3m48s) kubelet Back-off restarting failed container csi-resizer in pod hcloud-csi-controller-fb7688b79-2wql7_kube-system(a81a67c6-74d9-4baa-820d-977faec1fcaa)
Warning BackOff 3m16s (x4 over 3m48s) kubelet Back-off restarting failed container csi-provisioner in pod hcloud-csi-controller-fb7688b79-2wql7_kube-system(a81a67c6-74d9-4baa-820d-977faec1fcaa)
Warning BackOff 3m15s (x6 over 3m48s) kubelet Back-off restarting failed container csi-attacher in pod hcloud-csi-controller-fb7688b79-2wql7_kube-system(a81a67c6-74d9-4baa-820d-977faec1fcaa)
Warning Unhealthy 3m15s (x10 over 3m43s) kubelet Liveness probe failed: Get "http://10.244.3.3:9808/healthz": dial tcp 10.244.3.3:9808: connect: connection refused

[lukasmetzner]

@angustohrallegrinski

only private networks (--apiserver-advertise-address=xx on init)

Can you reach api.hetzner.cloud from inside your cluster?

coredns does not start until I deply hetzner ccm-networks

Is your cluster in a healthy state?

You can try setting the environment variable LOG_LEVEL to debug to get more output. Additionally, you can try setting the environment variable HCLOUD_DEBUG to true. This should result in every API request being logged.

If you post log messages here, please try using Markdown code blocks for better readability: https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks

[lefterisALEX]

experienced same issue and noticed that when i logged in to the master node where hcloud-csi-controller pod was running had no internet access ( using private nodes).
When i fixed it pod came up directly

[codablock]

I can also confirm that broken connectivity in hcloud-csi-controller was the root cause for the same behaviour. Maybe the controller should always print connectivity errors in the log no matter which log level is set...would make spotting the underlying issue much easier :)

[angustohrallegrinski]

It looked to me like the container didn't have DNS resolution capabilities. My cluster listens on private IPs, but the machines also have public IPs and aren't having any connectivity issues.
@lefterisALEX , how did you fix it? Your node didn't have any connectivity and you gave it to him? But that's not my case.
@codablock Yes, I think the container checks something about the connectivity on the entrypoint itself, which is why the logs are then lost. These checks should probably be done on cmd.

[lukasmetzner]

In v2.20.0 we have added two info logs during startup, so logs are not empty anymore until the first volume operation is processed. The second log message prints the configured default volume location. To evaluate this, depending on the configuration this might use an API request and could be an indication for any networking issues.

v2.19.1 also ships with a default timeout for the http client used to connect to the Hetzner Cloud API. This should increase visibility, once requests receive a timeout.

[lukasmetzner]

@angustohrallegrinski Can you maybe try this with the newer CSI-Driver version v2.20.0? If this is really a DNS connectivity issue on your side, I would close this ticket, as it is unrelated.