Running KMS plugin as a static pod

[ArteChp]

Hello,
What's the better way to run a KMS Plugin in Talos?
I'm trying to run and set up KMS Plugin for Infisical as a static pod in Talos 1.14.0-beta.0
but I can't find a solution to mount a socket from the static pod to the pod of API server through the host filesystem.
API server logs:

E0730 15:26:48.134852 1 grpc_service.go:70] "failed to create connection to unix socket" err="dial unix /var/run/infisical/infisicalkms.socket: connect: permission denied" addr="/var/run/infisical/infisicalkms.socket"

Modification of the machine config:

  pods:
       - apiVersion: v1
         kind: Pod
         metadata:
           name: infisical-kms-plugin
           namespace: kube-system
           labels:
             tier: control-plane
             component: infisical-kms-plugin
         spec:
           priorityClassName: system-node-critical
           hostNetwork: true
           containers:
             - name: infisical-kms-plugin
               image: infisical/kubernetes-kms:v0.0.2
               imagePullPolicy: IfNotPresent
               args:
                 - --host-url=http://192.168.x.x:8080
                 - --ua-client-id=<ua-client-id-here>
                 - --ua-client-secret=<ua-client-secret-here>
                 - --kms-key=<kms-key-id-here>
               securityContext:
                 allowPrivilegeEscalation: false
                 capabilities:
                   drop:
                     - ALL
                 readOnlyRootFilesystem: true
                 runAsUser: 0
               ports:
                 - containerPort: 8787 # Must match the value defined in --healthz-port
                   protocol: TCP
               livenessProbe:
                 httpGet:
                   path: /healthz # Must match the value defined in --healthz-path
                   port: 8787 # Must match the value defined in --healthz-port
                 failureThreshold: 2
                 periodSeconds: 10
               # resources:
               #   requests:
               #     cpu: 100m
               #     memory: 128Mi
               #   limits:
               #     cpu: 4
               #     memory: 2Gi
               volumeMounts:
                 - name: etc-kubernetes
                   mountPath: /etc/kubernetes
                 - name: sock
                   mountPath: /opt
           volumes:
             - name: etc-kubernetes
               hostPath:
                 path: /etc/kubernetes
             - name: sock
               hostPath:
                 path: /var/run/infisical

   apiServer:
       extraVolumes:
           - hostPath: /var/run/infisical/infisicalkms.socket
             mountPath: /var/run/infisical/infisicalkms.socket
             readonly: false
---
apiVersion: v1alpha1
kind: KubeEtcdEncryptionConfig
config:
   resources:
       - providers:
           - identity: {}
           - kms:
               apiVersion: v2
               name: infisical-kms-plugin
               endpoint: unix:///var/run/infisical/infisicalkms.socket    # This should match the listen-addr declared in the Infisical KMS plugin's static pod definition
               timeout: 20s
         resources:
           - secrets

How to resove the probem with the permission? Is it possible to run the plugin as a static pod or perhaps system extension better?

[smira]

I don't think it's a great idea to run it this way - seems fragile to me.

But your problem I guess is that apiserver runs as non-root, and the socket is created by root-owned static pod, so you need match users.

[ArteChp]

Thanks for the advice!
Yeah, I thought about a matching users and found that the container image of api server built with a root user:

$ docker image inspect registry.k8s.io/kube-apiserver:v1.36.2
...
        "Config": {
            "User": "0",
...

Also I reviewed how the kubernetes runs the api server and didn't find some instructions for users:

$ kubectl get pod kube-apiserver-talos-pda-cxz -n kube-system -o jsonpath='{.spec.containers[*].securityContext.runAsUser}'

And the container registry.k8s.io/kube-apiserver:v1.36.2 doesn't contain a shell.

So I'll continue research.

Is a system extension better for the plugin? But I couldn't find how to set up healthchecks of an extension service exactly in the documentation.

[rakeshutekar]

The connect: permission denied (EACCES) on the socket is the key detail — it's different from "no such file" or "connection refused." The socket exists and the path is reachable, but the kube-apiserver process isn't allowed to open it. That almost always comes down to socket ownership/mode versus the user the apiserver runs as, plus making sure both static pods share the exact same host directory.

1. Share one host directory between both static pods

The KMS plugin and kube-apiserver are two separate static pods, so the socket has to live on a hostPath that both mount:

• KMS plugin pod (your machine.pods entry): mount /var/run/infisical as a hostPath and have Infisical create/listen on …/infisicalkms.socket there.

• kube-apiserver: Talos manages it, so you add the same directory via machine config rather than editing the manifest:

  cluster:
    apiServer:
      extraVolumes:
        - name: infisical-kms
          hostPath: /var/run/infisical
          mountPath: /var/run/infisical
          readonly: false
      extraArgs:
        encryption-provider-config: /etc/kubernetes/encryptionconfig.yaml

  with the EncryptionConfiguration pointing at endpoint: unix:///var/run/infisical/infisicalkms.socket. (/var/run is tmpfs on Talos, so this is writable despite the read-only rootfs.)

If the apiserver mounts a different path than where the plugin actually created the socket, you'd get "no such file" — since you're getting permission denied, the mount is right and it's the next point.

2. Make the socket accessible to the apiserver's user

Talos runs kube-apiserver as a non-root user, while your Infisical static pod almost certainly runs as root and creates the socket with mode 0600 (owner-only). Root's 0600 socket is unreadable to the apiserver's UID → EACCES on connect. Fixes, in order of preference:

1. Have the plugin create the socket world/group-accessible. Many KMS plugins expose a flag or honor umask; set the listen socket to 0660 or 0666. This is the cleanest fix.
2. Run the plugin as the same UID as the apiserver. Add a securityContext to the Infisical static pod with runAsUser/runAsGroup matching the apiserver, so the socket it creates is owned by that user.
3. Loosen the directory. Ensure /var/run/infisical itself is traversable (0755) and, if you go the shared-group route, that both pods share a GID with group-execute on the dir.

To find the UID the apiserver actually runs as, check the generated static pod:

talosctl -n <node> get staticpods kube-apiserver -o yaml

and look at the container securityContext (runAsUser) — that's the UID that must be able to open the socket.

3. Ordering (usually not the cause of EACCES, but worth confirming)

The plugin must be up before the apiserver tries to use it. Talos will restart the apiserver until the KMS endpoint answers, so a brief crash-loop at boot is normal — but that surfaces as connection-refused, not permission-denied, so it's not your current symptom.

My money is on #2 (socket mode): a root-owned 0600 socket vs a non-root apiserver is the textbook cause of exactly this error with gRPC KMS plugins. If you paste the Infisical static pod's securityContext (and whether it exposes a socket-permission/umask option), I can point at the precise line to change. If you'd rather sidestep per-file perms entirely, running the plugin with runAsUser equal to the apiserver's UID is the most deterministic fix.

[ArteChp]

rakeshutekar, thank you!

The command talosctl -n <node> get staticpods kube-apiserver -o yaml helped to get uid and gid of the api server.

I've tried to change runAsUser and runAsGroup in static pod of the plugin:

pods:
     - apiVersion: v1
       kind: Pod
       metadata:
         name: infisical-kms-plugin
         namespace: kube-system
         labels:
           tier: control-plane
           component: infisical-kms-plugin
       spec:
         priorityClassName: system-node-critical
         hostNetwork: true
         containers:
           - name: infisical-kms-plugin
             image: infisical/kubernetes-kms:v0.0.2
             imagePullPolicy: IfNotPresent
             args:
               - --host-url=http://192.168.x.x:8080
               - --ua-client-id=<ua-client-id-here>
               - --ua-client-secret=<ua-client-secret-here>
               - --kms-key=<kms-key-id-here>
               - --listen-addr=/var/run/infisical/infisicalkms.socket
             securityContext:
               allowPrivilegeEscalation: false
               capabilities:
                 drop:
                   - ALL
               readOnlyRootFilesystem: true
               # runAsUser: 0
               runAsUser: 65534
               runAsGroup: 65534
             ports:
               - containerPort: 8787 # Must match the value defined in --healthz-port
                 protocol: TCP
             livenessProbe:
               httpGet:
                 path: /healthz # Must match the value defined in --healthz-path
                 port: 8787 # Must match the value defined in --healthz-port
               failureThreshold: 2
               periodSeconds: 10
             # resources:
             #   requests:
             #     cpu: 100m
             #     memory: 128Mi
             #   limits:
             #     cpu: 4
             #     memory: 2Gi
             volumeMounts:
               - name: etc-kubernetes
                 mountPath: /etc/kubernetes
               - name: sock
                 mountPath: /var/run/infisical
         volumes:
           - name: etc-kubernetes
             hostPath:
               path: /etc/kubernetes
           - name: sock
             hostPath:
               path: /var/run/infisical

and also changed the mount in machine config:

    apiServer:
        extraVolumes:
            - hostPath: /var/run/infisical
              mountPath: /var/run/infisical
              readonly: false

but the plugin doesn't work with these user and group.

2026/07/31 08:37:46 Using auth method: UNIVERSAL_MACHINE_IDENTITY
2026/07/31 08:37:46 Starting up health check server...
2026/07/31 08:37:46 Starting up plugin server...
2026/07/31 08:37:46 Error running services: listen unix /var/run/infisical/infisicalkms.socket: bind: permission denied

It seems the user doesn't have permissions for the folder /var/run/infisical/ on the host (Talos filesystem), right?
Is it possible to change permissions of the folder in Talos machine config?

[rakeshutekar]

Nice, you're basically there. Notice the error changed from connect: permission denied to bind: permission denied — that's actually the plugin now failing to create the socket rather than the apiserver failing to open it. Makes sense: the kubelet creates that hostPath dir as root (0755), so once you drop the plugin to 65534 it can't write into /var/run/infisical anymore.

Matching the UID to the apiserver is still the right call though — it quietly fixes the connect side too, since a socket created by 65534 is owned by 65534, and the apiserver (also 65534) can always connect to a socket it owns no matter what mode it ends up with. So you don't actually need that 0660/0666 flag you were about to ask Infisical for. You just need the dir to be writable by 65534 so the bind works.

fsGroup won't help here (kubelet doesn't apply it to hostPath volumes), so the simplest thing is a little init container that chowns the dir before the plugin starts:

spec:
  initContainers:
    - name: prepare-socket-dir
      image: busybox:1.36        # kms image is probably distroless, so use something with a shell
      command: ["sh","-c","mkdir -p /var/run/infisical && chown 65534:65534 /var/run/infisical"]
      securityContext:
        runAsUser: 0
        capabilities:
          drop: ["ALL"]
          add:  ["CHOWN"]
      volumeMounts:
        - name: sock
          mountPath: /var/run/infisical
  # containers: unchanged, still runAsUser/runAsGroup 65534
  volumes:
    - name: sock
      hostPath:
        path: /var/run/infisical
        type: DirectoryOrCreate

Once that runs, the dir belongs to 65534, the plugin binds fine, and the apiserver connects fine. And since /var/run is tmpfs it gets wiped on reboot, but the init container just re-runs and re-chowns it, so it's self-healing.

Re: doing it from the Talos side — I wouldn't. /var/run is tmpfs so machine config isn't really the place to manage that dir (and yeah, files[] is on its way out in 1.14 anyway). Keeping it in the pod is less to reason about.

And +1 to what @smira said — a socket on the host fs is a bit awkward however you slice it, so if the investigation turns up a nicer pattern that's probably the better home for this. The above is just to get you unblocked in the meantime.

[ArteChp]

It works! Thank you!

[ArteChp]

Have the plugin create the socket world/group-accessible. Many KMS plugins expose a flag or honor umask; set the listen socket to 0660 or 0666. This is the cleanest fix.

It's clear now, thanks!
It would be great but there aren't settings in the plugin to set the listen socket to 0660 or 0666.
I'll try to create an issue for the feature request if will not find another solution.

[ArteChp]

Loosen the directory. Ensure /var/run/infisical itself is traversable (0755) and, if you go the shared-group route, that both pods share a GID with group-execute on the dir.

How to implement it in Talos 1.14.0-beta.0 ?
files[] is deprecated.

[smira]

Let's put this a bit on hold and do some investigation. Using file socket via the filesystem doesn't seem to be really great solution either way.

[ArteChp]

Sure, what about running the KMS plugin as a sidecar container for the API server and creating the socket in an emptyDir: {} volume?