`enableProvidedByTopology` can provision volumes in the wrong location

[M4t7e]

TL;DR

With global.enableProvidedByTopology=true and a WaitForFirstConsumer StorageClass, volumes can be provisioned in the hcloud CSI controller pod's location instead of the location of the node selected for the workload.

The generated allowedTopologies contains only instance.hetzner.cloud/provided-by=cloud. Because csi-provisioner runs without --strict-topology, the selected node's csi.hetzner.cloud/location is omitted from the CreateVolume request.

Expected behavior

When delayed binding selects a node for a PVC-backed pod, the volume should be created in that node's Hetzner location.

The CreateVolume accessibility requirements should include the selected node's complete CSI topology, including csi.hetzner.cloud/location, while instance.hetzner.cloud/provided-by=cloud continues to prevent provisioning on Robot nodes.

Observed behavior

With global.enableProvidedByTopology=true, the generated StorageClass restricts provisioning using only:

allowedTopologies:
  - matchLabelExpressions:
      - key: instance.hetzner.cloud/provided-by
        values:
          - cloud

The Helm chart starts csi-provisioner without --strict-topology: https://github.com/hetznercloud/csi-driver/blob/v2.21.2/chart/templates/controller/deployment.yaml#L193-L206

Without strict topology, external-provisioner uses allowedTopologies as the requisite topology and does not add the selected node's missing csi.hetzner.cloud/location: https://github.com/kubernetes-csi/external-provisioner/blob/v6.2.0/pkg/controller/topology.go#L215-L239

The hcloud CSI controller initializes the volume location with the controller pod's location and only overrides it when csi.hetzner.cloud/location is present in the accessibility requirements: https://github.com/hetznercloud/csi-driver/blob/v2.21.2/internal/driver/controller.go#L82-L88

The resulting PV can therefore be created in a different location than the node selected for the pod. The pod then remains unschedulable because the PV node affinity does not match the selected node.

Minimal working example

1. Run Hetzner Cloud worker nodes in at least two locations.
2. Enable global.enableProvidedByTopology.
3. Use the generated StorageClass with volumeBindingMode: WaitForFirstConsumer.
4. Deploy multiple PVC-backed replicas with topology spread constraints so that Kubernetes selects nodes across the locations:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: topology-repro
   spec:
     replicas: 2
     selector:
       matchLabels: {app: topology-repro}
     template:
       metadata:
         labels: {app: topology-repro}
       spec:
         topologySpreadConstraints:
         - maxSkew: 1
           topologyKey: topology.kubernetes.io/region
           whenUnsatisfiable: DoNotSchedule
           labelSelector:
             matchLabels: {app: topology-repro}
         containers:
         - name: pause
           image: registry.k8s.io/pause:3.10
         volumes:
         - name: data
           ephemeral:
             volumeClaimTemplate:
               spec:
                 accessModes: [ReadWriteOnce]
                 resources:
                   requests:
                     storage: 10Gi

Log output

❯ kubectl get pods,pvc -l app=topology-repro
NAME                                 READY   STATUS    RESTARTS   AGE
pod/topology-repro-54bb978c4-8l6v4   1/1     Running   0          8s
pod/topology-repro-54bb978c4-lc572   0/1     Pending   0          8s

❯ kubectl get events --field-selector involvedObject.kind=Pod,involvedObject.name=topology-repro-54bb978c4-lc572 --sort-by=.lastTimestamp
LAST SEEN   TYPE      REASON             OBJECT                               MESSAGE
27s         Warning   FailedScheduling   pod/topology-repro-54bb978c4-lc572   0/11 nodes are available: waiting for ephemeral volume controller to create the persistentvolumeclaim "topology-repro-54bb978c4-lc572-data". not found
17s         Warning   FailedScheduling   pod/topology-repro-54bb978c4-lc572   running PreBind plugin "VolumeBinding": binding volumes: pv "pvc-d723297e-20f8-4c06-8406-f330799943a7" node affinity doesn't match node "alpha-prd-k8s-worker-fsn1-b-1": no matching NodeSelectorTerms
16s         Warning   FailedScheduling   pod/topology-repro-54bb978c4-lc572   0/11 nodes are available: 3 node(s) had untolerated taint(s), 4 node(s) didn't match PersistentVolume's node affinity, 4 node(s) didn't match pod topology spread constraints. no new claims to deallocate, preemption: 0/11 nodes are available: 4 No preemption victims found for incoming pod, 7 Preemption is not helpful for scheduling.

❯ kubectl get pv -o jsonpath='{range .items[*]}{.spec.claimRef.name}{"\t"}{.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[?(@.key=="csi.hetzner.cloud/location")].values[*]}{"\n"}{end}' | grep topology-repro
topology-repro-54bb978c4-8l6v4-data	nbg1
topology-repro-54bb978c4-lc572-data	nbg1

Additional information

There are two possible fixes:

1. Enable --strict-topology when global.enableProvidedByTopology=true, so the complete topology of the scheduler-selected node is passed to the CSI driver:

   {{- if .Values.global.enableProvidedByTopology }}
   - --strict-topology
   {{- end }}

   Introduce new flag - strict-topology kubernetes-csi/external-provisioner#282 introduced --strict-topology to ensure delayed-bound volumes are created in the topology of the scheduler-selected node. The same safeguard would prevent this issue, where allowedTopologies omits csi.hetzner.cloud/location.

2. Include csi.hetzner.cloud/location in the generated allowedTopologies, so the selected node's location is passed to the CSI driver:

   allowedTopologies:
     - matchLabelExpressions:
         - key: instance.hetzner.cloud/provided-by
           values: [cloud]
         - key: csi.hetzner.cloud/location
           values: [fsn1, nbg1, hel1, ash, hil, sin]

   This alternative requires the location list to be kept up to date.

I tested both options successfully.

Likely related issues:

• CSI Controller will only create volumes in its own region again #822
• enableProvidedByTopology checks wrong label? #931