Raspberry Pi5 control plane ethernet issue

[ram4444]

Bug Report

Hi, I have tried to install Talos to RPI 5 (which finally support recently) as a control plan. It is ok to setup the cluster but I still found it will cut off the internet connection at a random time interval. In the UART sometimes I found error message like DNS not reachable. And after disable something the error message changed to failed refreshing discovery service data or simply timeserver not found. And I found the RPi5 is not found in my router's DHCP. Both talosctl and kubectl not work.

Description

Logs

Environment

• Talos version: [talosctl version --nodes <problematic nodes>]
• Kubernetes version: [kubectl version] v1.35.0
• Platform: SBC Raspberry Pi5

Installed by talosctl gen config with tailscale config
and with patches afterwards

apiVersion: v1alpha1 kind: ResolverConfig nameservers: - address: 8.8.8.8 - address: 1.1.1.1 - address: 192.168.0.1

machine: network: nameservers: - 8.8.8.8 - 1.1.1.1 - 192.168.0.1 features: hostDNS: enabled: false

[Alphaegen]

I am also seeing repeatable control-plane network loss on a fresh Talos cluster after installing the new rpi5 Talos v1.12.4 version on 4 Raspberry Pi 5. (following Dsseng's instructions in siderolabs/talos#12748)

They're powered by the POE of the NVME HAT.

What happens:

• control-plane IP cannot be pinged
• Talos API (:50000) times out / host down
• Kubernetes API becomes unreachable
• worker nodes remain reachable (ping)
• NIC LEDs on control-plane still show link/activity

Environment

• Cluster: 1 control-plane + 3 workers
• Hardware: 4x Raspberry Pi 5 with M.2 PoE HAT
• Talos: v1.12.4
• Kubernetes: v1.34.4
• Services installed when reproducing (roughly in order):
  • Cilium
  • MetalLB
  • cert-manager
  • Argo CD

Image used

Built locally (RPi5 overlay), with extensions:

docker run --rm -t -v ./_out:/out -v /dev:/dev --privileged \
    ghcr.io/siderolabs/imager:v1.12.4 rpi_5 \
    --base-installer-image="ghcr.io/siderolabs/installer-base:v1.12.4" \
    --overlay-image="ghcr.io/siderolabs/sbc-raspberrypi:v0.1.8-2-ge56ea57" \
    --overlay-name="rpi_5" \
    --arch arm64 \
    --system-extension-image=ghcr.io/siderolabs/gvisor:20260202.0 \
    --system-extension-image=ghcr.io/siderolabs/iscsi-tools:v0.2.0 \
    --system-extension-image=ghcr.io/siderolabs/util-linux-tools:2.41.2

Reproduction

1. Install fresh Talos v1.12.4 cluster.
2. Bootstrap and bring up CP.
3. Apply core services (Cilium, MetalLB, cert-manager, Argo CD).
4. After some time, control-plane network drops.

Last captured failure window

From the current capture session:

• CP healthy/running until 2026-02-23T00:47:34Z
• First down sample at 2026-02-23T00:47:36Z
• Talos dmesg stream immediately starts timing out on 192.168.3.101:50000

Possibly relevant lines:

=== dmesg disconnect rc=0 2026-02-23T00:47:34Z ===
=== dmesg connect 2026-02-23T00:47:35Z ===
error getting dmesg: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 192.168.3.101:50000: i/o timeout"

Logs:

summary log:
cp-drop-evidence-brief-20260223.log

dmesg:
cp-192-168-3-101-dmesg-20260223-014107.log

state:
cp-192-168-3-101-state-20260223-014144.log
cp-192-168-3-101-state-20260223-014107.log

serial:
cp-serial-20260223-014107.log

Sorry for any rough formatting or wrong explanations here, as I'm very new to both debugging and reporting on issues like these. If I need to add any more info feel free to tell me what's missing, as I've been debugging this for hours but can't find a cause for the issue.

[Alphaegen]

This is how my kubectl get pods -A looks like literally seconds before crashing, which I can reproduce now by rebooting the controlplane with the aforementioned services:

❯ kubectl get pods -A
NAMESPACE          NAME                                                READY   STATUS              RESTARTS        AGE
argocd             argocd-application-controller-0                     1/1     Running             0               14m
argocd             argocd-applicationset-controller-5ddbc7d89b-fhpcw   1/1     Running             0               8m29s
argocd             argocd-applicationset-controller-5ddbc7d89b-vt6bl   1/1     Running             0               8m29s
argocd             argocd-dex-server-5db4676df4-2w99s                  1/1     Running             0               8m29s
argocd             argocd-notifications-controller-64d78958c4-78mlx    1/1     Running             0               8m29s
argocd             argocd-redis-ha-haproxy-8467cfb56-b8g5t             0/1     Init:0/1            0               8m29s
argocd             argocd-redis-ha-haproxy-8467cfb56-l9h96             0/1     Init:Error          5 (3m24s ago)   27m
argocd             argocd-redis-ha-haproxy-8467cfb56-q7dv5             0/1     Terminating         5 (3m4s ago)    28m
argocd             argocd-redis-ha-haproxy-8467cfb56-zfb6m             0/1     Pending             0               8m19s
argocd             argocd-redis-ha-server-0                            0/3     Init:0/1            6 (3m8s ago)    14m
argocd             argocd-redis-ha-server-1                            0/3     Terminating         6 (4m ago)      33m
argocd             argocd-repo-server-85994c8dcc-9xvs4                 0/1     Running             0               8m19s
argocd             argocd-repo-server-85994c8dcc-fq7mz                 1/1     Running             0               8m29s
argocd             argocd-server-7c45dd656c-lwtm5                      1/1     Running             0               8m19s
argocd             argocd-server-7c45dd656c-wv6p5                      1/1     Running             0               8m29s
cert-manager       cert-manager-8455fdbf67-nvtcg                       1/1     Running             0               8m30s
cert-manager       cert-manager-cainjector-77dbd66479-dzgnv            1/1     Running             0               8m19s
cert-manager       cert-manager-webhook-78dff5f574-qpczt               1/1     Running             0               8m29s
external-secrets   external-secrets-7fcdbd8d79-fm984                   1/1     Running             0               8m19s
external-secrets   external-secrets-cert-controller-7b7858f6f-x96f8    0/1     Running             0               8m19s
external-secrets   external-secrets-webhook-66f565986c-pt7vd           0/1     Running             0               8m29s
ingress-nginx      nginx-ingress-nginx-controller-79f686955f-4rbsl     0/1     ContainerCreating   0               8m29s
kube-system        cilium-54ck5                                        1/1     Running             3 (56y ago)     53m
kube-system        cilium-c77kd                                        1/1     Running             0               53m
kube-system        cilium-envoy-nhql5                                  1/1     Running             1 (56y ago)     53m
kube-system        cilium-envoy-psf2l                                  1/1     Running             3 (56y ago)     53m
kube-system        cilium-envoy-sg5b5                                  1/1     Running             0               53m
kube-system        cilium-envoy-xvqdd                                  1/1     Running             1 (56y ago)     53m
kube-system        cilium-lbp8x                                        1/1     Running             0               53m
kube-system        cilium-operator-848d4987f9-9ssbm                    1/1     Running             1 (56y ago)     28m
kube-system        cilium-operator-848d4987f9-jt48k                    1/1     Running             0               8m29s
kube-system        cilium-rgqtc                                        1/1     Running             0               53m
kube-system        coredns-76899f5fd7-ffqft                            0/1     ContainerCreating   0               27m
kube-system        coredns-76899f5fd7-fgwqj                            0/1     ContainerCreating   0               27m
kube-system        hubble-relay-6cdd9bfb76-jxt6d                       0/1     Running             0               8m19s
kube-system        hubble-ui-ffdc7bfb5-5phsf                           2/2     Running             0               8m19s
kube-system        kube-apiserver-rpi-cp-1                             1/1     Running             0               23s
kube-system        kube-controller-manager-rpi-cp-1                    1/1     Running             1 (75s ago)     23s
kube-system        kube-scheduler-rpi-cp-1                             1/1     Running             1 (76s ago)     23s
metallb-system     metallb-controller-765c495b75-bzkqw                 0/1     Running             0               8m19s

[smira]

I moved the issue to the proper repository.

[Alphaegen]

I moved the issue to the proper repository.

Ah thank you. I wasn't sure whether it was suposed to be created here or there.

[Alphaegen]

I think it's also important to mention that this exact setup worked fine on the custom Talos 1.10.7 rpi5 setup.

I remember someone building a patch to make that version of Talos work on rpi5's, which did work did for months. When attempting to now recreate my cluster from scratch I'm running into these issues.

[charliebravo636]

Same issue here. Everything worked fine when i deployed Cilium, after i deployed Longhorn the Cluster comes up successfully, but after a few minutes the ethernet connection to the control plane is lost. There is no dmesg output that anything fails before the connection failed. Seems like too much load on the control plane is creating this.

[charliebravo636]

https://bugs.launchpad.net/ubuntu/+source/linux-raspi/+bug/2133877, this issue seemed to be the same, so i applied this Patch to the Machineconfig:

machine:
install:
extraKernelArgs:
- cpufreq.default_governor=performance

After this i had no issues so far. But it is still just a workaround, and still not sure if this is stable.
Ok the problem occurs again after 2 hours. There was also no higher traffic than before.

[Reversing0148]

I bumped into the same issue on my end:

• 4 x Raspberry Pi CM5108016 (1cp, 3wo)
• talos 1.12.4 with --overlay-image="ghcr.io/siderolabs/sbc-raspberrypi:v0.1.8-2-ge56ea57"
• kubernetes v1.35.0
• metrics-server latest
• kubelet-serving-cert-approver latest
• cilium 1.19.1
• ceph tentacle

By the time the ceph-cluster finished initializing (~20 minutes) the control plane dropped from the network. Same symptoms as above.

[charliebravo636]

It seems to be enough to restart the Interface, after that the issue is gone for a while. I made a DaemonSet on the control plane that restarts the interface when a ping to my firewall is not working.

[ram4444]

As I found there is release, I use the latest --schematic-id=6c2662f97725518ef68c879e5210a1a0367c5d8eff3c76e89d0f41da847e05d1
to UPGEADE but the issue still remains

[nuno99costa]

Same issue here. For now I have also added a daemon set to restart the network interface on my router not being reachable

[Alphaegen]

Just a reminder for everyone here. You can upgrade to ghcr.io/siderolabs/sbc-raspberrypi:v0.2.0 if you're still on 0.1.8 as 0.1.9 seems to have some changes that help with the cluster breaking less. It still has network issues tho.

[SeanSith]

I'm having the same issue.

My cluster:

• 4x Pi 5, 8GB RAM
• 128GB SD card for boot, 500GB SSD via USB for storage.
• powered with a Waveshare PoE HAT (F).
• Static IP assigned to the control-panel node, but normally it's a DHCP provided static assignment.
• This is a vanilla cluster with no pods/deployments that I was setting up to learn Talos and K8s.

All of this hardware had previously worked flawlessly on Raspbian 13 and running k3s.

Symptoms:

• Physical link stays up during dropout.
• Node remains powered on via PoE.
• Node continues running, console is active as viewed with connected monitor.
• No kernel messages at time of dropout aside from continued timeouts to DNS and the time controller.

I've tried to build a custom image with the following schematic:

overlay:
    image: siderolabs/sbc-raspberrypi:v0.2.0
    name: rpi_5
    options:
        configTxtAppend: |
            usb_max_current_enable=1
customization:
    systemExtensions:
        officialExtensions:
            - siderolabs/iscsi-tools
            - siderolabs/util-linux-tools
            - siderolabs/vc4

but I can't really tell if the control-plane node is running v0.2.0 or v0.1.8 as while the output from the build looks like

profile ready:
arch: arm64
platform: metal
secureboot: false
version: v1.12.4
input:
  kernel:
    path: /usr/install/arm64/vmlinuz
  initramfs:
    path: /usr/install/arm64/initramfs.xz
  sdStub:
    path: /usr/install/arm64/systemd-stub.efi
  sdBoot:
    path: /usr/install/arm64/systemd-boot.efi
  baseInstaller:
    imageRef: ghcr.io/siderolabs/installer-base:v1.12.4
  overlayInstaller:
    imageRef: ghcr.io/siderolabs/sbc-raspberrypi:v0.2.0
  systemExtensions:
    - imageRef: ghcr.io/siderolabs/vc4:v1.12.4
    - imageRef: ghcr.io/siderolabs/iscsi-tools:v0.2.0
    - imageRef: ghcr.io/siderolabs/util-linux-tools:2.41.2
overlay:
  name: rpi_5
  image:
    imageRef: ghcr.io/siderolabs/sbc-raspberrypi:v0.2.0
  options:
    configTxtAppend: usb_max_current_enable=1
output:
  kind: image
  imageOptions:
    diskSize: 2355101696
    diskFormat: raw
    bootloader: grub
  outFormat: .xz

but talosctl image talos-bundle lists:

...
ghcr.io/siderolabs/sbc-raspberrypi:v0.1.8@sha256:958f671aecd3c042dba359f02dae61c0ef32867fee200c94239b1d5a8c6d122c
...

I can cause the problem using talosctl conformance kubernetes, but it will eventually timeout on its own, too.