PCI BAR Memory Allocation Issue on RK3568-based Device NanoPi R5S

[eggcllnt]

Hardware/Software Environment

Hardware: NanoPi with Rockchip RK3568 processor

Issue Description
While working with my RK3568-based device, I observed errors related to PCI BAR memory allocation on one of the PCI bridges. The system appears to have three PCIe controllers, but the third one (0002:00:00.0) is showing error messages related to memory allocation.

dmesg Output

[    3.584908] pci_bus 0002:00: root bus resource [bus 00-0f]
[    3.585420] pci_bus 0002:00: root bus resource [io  0x200000-0x2fffff] (bus address [0xf0100000-0xf01fffff])
[    3.586307] pci_bus 0002:00: root bus resource [mem 0xf0200000-0xf1ffffff]
[    3.586935] pci_bus 0002:00: root bus resource [mem 0x380000000-0x3bfffffff] (bus address [0x40000000-0x7fffffff])
[    3.587998] pci 0002:00:00.0: [1d87:3566] type 01 class 0x060400
[    3.588571] pci 0002:00:00.0: reg 0x10: [mem 0x00000000-0x3fffffff]
[    3.589151] pci 0002:00:00.0: reg 0x14: [mem 0x00000000-0x3fffffff]
[    3.589730] pci 0002:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    3.590420] pci 0002:00:00.0: supports D1 D2
[    3.590819] pci 0002:00:00.0: PME# supported from D0 D1 D3hot
[    3.599827] pci_bus 0002:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
[    3.601189] pci 0002:00:00.0: BAR 0: assigned [mem 0x380000000-0x3bfffffff]
[    3.601846] pci 0002:00:00.0: BAR 1: no space for [mem size 0x40000000]
[    3.602454] pci 0002:00:00.0: BAR 1: failed to assign [mem size 0x40000000]
[    3.603088] pci 0002:00:00.0: BAR 6: assigned [mem 0xf0200000-0xf020ffff pref]
[    3.603811] pci 0002:00:00.0: PCI bridge to [bus 01-ff]
[    3.607148] pcieport 0002:00:00.0: PME: Signaling with IRQ 75
[    3.608250] pcieport 0002:00:00.0: AER: enabled with IRQ 75

lspci Output

root@DietPi:~# lspci
0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)
0000:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0001:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)
0001:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0002:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)

Memory Map Information

300000000-33fffffff : pcie@fe260000
340000000-37fffffff : pcie@fe270000
380000000-3bfffffff : pcie@fe280000
  380000000-3bfffffff : 0002:00:00.0
3c0000000-3c03fffff : 3c0000000.pcie dbi
3c0400000-3c07fffff : 3c0400000.pcie dbi
3c0800000-3c0bfffff : 3c0800000.pcie dbi

The logs show:

The system has three PCIe controllers
Each controller is allocated 1GB of memory space
Controller at 0002:00:00.0 successfully assigns BAR 0 to memory range 0x380000000-0x3bfffffff
Controller at 0002:00:00.0 fails to assign BAR 1 which requires 0x40000000 (1GB) of memory
There's also a bus number allocation conflict: "can not insert [bus 01-ff] under [bus 00-0f]"

Steps Taken

Observed the issue during system boot
Collected dmesg logs showing PCI resource allocation
Ran lspci to check detected PCI devices
Checked memory mapping via /proc/iomem

Questions

Is this a known limitation of the RK3568 PCIe controller configuration?
Are there any known workarounds such as kernel parameters or device tree modifications?
Could this indicate a firmware/BIOS issue that might be addressed in a future update?

Expected behavior

The NVMe drive should be consistently detected as /dev/nvme0n1 after booting, allowing me to install an operating system on it or use it for storage.

Additional information

The same NVMe drive works consistently with FriendlyElec's official Debian image, so the issue appears to be related to DietPi's kernel configuration or U-Boot settings rather than a hardware problem. I've also observed that in rare cases, the NVMe drive does get detected after a reboot, but this behavior is inconsistent.
Key error messages from the PCIe subsystem:

busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
BAR 1: no space for [mem size 0x40000000]
BAR 1: failed to assign [mem size 0x40000000]

I would appreciate assistance in resolving this issue to allow for consistent NVMe detection and usage on the NanoPi R5S.

[inindev]

What kernel is running?

uname -a

[bdew]

FWIW i'm also seeing those errors on my R5S systems (running kernel 6.14.4) and both have working NVMe drives with no issues.

[    2.077665] pci 0002:00:00.0: BAR 1 [mem size 0x40000000]: can't assign; no space
[    3.215169] pci_bus 0000:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])

[eggcllnt]

I decided fully reinstall system. No changes except changing password.

debian@bookworm-nanopi-r5s:~$ uname -a
Linux bookworm-nanopi-r5s 6.1.0-31-arm64 #1 SMP Debian 6.1.128-1 (2025-02-07) aarch64 GNU/Linux

debian@bookworm-nanopi-r5s:~$ lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
mmcblk1      179:0    0 28.9G  0 disk
└─mmcblk1p1  179:1    0 28.9G  0 part /
mmcblk1boot0 179:256  0    4M  1 disk
mmcblk1boot1 179:512  0    4M  1 disk

debian@bookworm-nanopi-r5s:~$ lspci -vv
0001:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 40
        Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
        I/O behind bridge: 1000-1fff [size=4K] [16-bit]
        Memory behind bridge: f2200000-f22fffff [size=1M] [32-bit]
        Prefetchable memory behind bridge: [disabled] [64-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        Expansion ROM at f2300000 [virtual] [disabled] [size=64K]
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: <access denied>
        Kernel driver in use: pcieport

0001:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 40
        Region 0: I/O ports at 1000 [size=256]
        Region 2: Memory at f2200000 (64-bit, non-prefetchable) [size=64K]
        Region 4: Memory at f2210000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: r8169
        Kernel modules: r8169

0002:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 43
        Region 0: Memory at 380000000 (32-bit, non-prefetchable) [size=1G]
        Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
        I/O behind bridge: [disabled] [16-bit]
        Memory behind bridge: [disabled] [32-bit]
        Prefetchable memory behind bridge: [disabled] [64-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        Expansion ROM at f0200000 [virtual] [disabled] [size=64K]
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: <access denied>
        Kernel driver in use: pcieport

debian@bookworm-nanopi-r5s:~$ sudo dmesg | grep -i -E 'pci|nvme'
[    0.170991] PCI: CLS 0 bytes, default 64
[    2.534101] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    5.689641] rockchip-dw-pcie 3c0400000.pcie: host bridge /pcie@fe270000 ranges:
[    5.690596] rockchip-dw-pcie 3c0400000.pcie:       IO 0x00f2100000..0x00f21fffff -> 0x00f2100000
[    5.693250] rockchip-dw-pcie 3c0400000.pcie:      MEM 0x00f2200000..0x00f3ffffff -> 0x00f2200000
[    5.694674] rockchip-dw-pcie 3c0400000.pcie:      MEM 0x0340000000..0x037fffffff -> 0x0040000000
[    5.701584] rockchip-dw-pcie 3c0400000.pcie: iATU unroll: enabled
[    5.702227] rockchip-dw-pcie 3c0400000.pcie: iATU regions: 8 ob, 8 ib, align 64K, limit 8G
[    5.911448] rockchip-dw-pcie 3c0400000.pcie: PCIe Gen.2 x1 link up
[    5.912359] rockchip-dw-pcie 3c0400000.pcie: PCI host bridge to bus 0001:00
[    5.913012] pci_bus 0001:00: root bus resource [bus 00-0f]
[    5.913522] pci_bus 0001:00: root bus resource [io  0x0000-0xfffff] (bus address [0xf2100000-0xf21fffff])
[    5.914383] pci_bus 0001:00: root bus resource [mem 0xf2200000-0xf3ffffff]
[    5.915006] pci_bus 0001:00: root bus resource [mem 0x340000000-0x37fffffff] (bus address [0x40000000-0x7fffffff])
[    5.916049] pci 0001:00:00.0: [1d87:3566] type 01 class 0x060400
[    5.916630] pci 0001:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    5.917316] pci 0001:00:00.0: supports D1 D2
[    5.917711] pci 0001:00:00.0: PME# supported from D0 D1 D3hot
[    5.923621] pci_bus 0001:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
[    5.924749] pci 0001:01:00.0: [10ec:8125] type 00 class 0x020000
[    5.925428] pci 0001:01:00.0: reg 0x10: initial BAR value 0x00000000 invalid
[    5.926235] pci 0001:01:00.0: reg 0x10: [io  size 0x0100]
[    5.926833] pci 0001:01:00.0: reg 0x18: [mem 0x00000000-0x0000ffff 64bit]
[    5.927571] pci 0001:01:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit]
[    5.928722] pci 0001:01:00.0: supports D1 D2
[    5.929128] pci 0001:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    5.939631] pci 0001:00:00.0: BAR 14: assigned [mem 0xf2200000-0xf22fffff]
[    5.940311] pci 0001:00:00.0: BAR 6: assigned [mem 0xf2300000-0xf230ffff pref]
[    5.940993] pci 0001:00:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    5.941586] pci 0001:01:00.0: BAR 2: assigned [mem 0xf2200000-0xf220ffff 64bit]
[    5.942313] pci 0001:01:00.0: BAR 4: assigned [mem 0xf2210000-0xf2213fff 64bit]
[    5.943028] pci 0001:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    5.943653] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    5.944144] pci 0001:00:00.0:   bridge window [io  0x1000-0x1fff]
[    5.944700] pci 0001:00:00.0:   bridge window [mem 0xf2200000-0xf22fffff]
[    5.947998] pcieport 0001:00:00.0: PME: Signaling with IRQ 40
[    5.949430] pcieport 0001:00:00.0: AER: enabled with IRQ 40
[    5.952124] rockchip-dw-pcie 3c0800000.pcie: host bridge /pcie@fe280000 ranges:
[    5.952864] rockchip-dw-pcie 3c0800000.pcie:       IO 0x00f0100000..0x00f01fffff -> 0x00f0100000
[    5.953691] rockchip-dw-pcie 3c0800000.pcie:      MEM 0x00f0200000..0x00f1ffffff -> 0x00f0200000
[    5.954542] rockchip-dw-pcie 3c0800000.pcie:      MEM 0x0380000000..0x03bfffffff -> 0x0040000000
[    5.962218] rockchip-dw-pcie 3c0800000.pcie: iATU unroll: enabled
[    5.962800] rockchip-dw-pcie 3c0800000.pcie: iATU regions: 8 ob, 8 ib, align 64K, limit 8G
[    7.075494] rockchip-dw-pcie 3c0800000.pcie: Phy link never came up
[    7.076666] rockchip-dw-pcie 3c0800000.pcie: PCI host bridge to bus 0002:00
[    7.077342] pci_bus 0002:00: root bus resource [bus 00-0f]
[    7.077851] pci_bus 0002:00: root bus resource [io  0x100000-0x1fffff] (bus address [0xf0100000-0xf01fffff])
[    7.078732] pci_bus 0002:00: root bus resource [mem 0xf0200000-0xf1ffffff]
[    7.079402] pci_bus 0002:00: root bus resource [mem 0x380000000-0x3bfffffff] (bus address [0x40000000-0x7fffffff])
[    7.080601] pci 0002:00:00.0: [1d87:3566] type 01 class 0x060400
[    7.081197] pci 0002:00:00.0: reg 0x10: [mem 0x00000000-0x3fffffff]
[    7.081774] pci 0002:00:00.0: reg 0x14: [mem 0x00000000-0x3fffffff]
[    7.082345] pci 0002:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    7.083184] pci 0002:00:00.0: supports D1 D2
[    7.083638] pci 0002:00:00.0: PME# supported from D0 D1 D3hot
[    7.089615] pci_bus 0002:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
[    7.090792] pci 0002:00:00.0: BAR 0: assigned [mem 0x380000000-0x3bfffffff]
[    7.091475] pci 0002:00:00.0: BAR 1: no space for [mem size 0x40000000]
[    7.092126] pci 0002:00:00.0: BAR 1: failed to assign [mem size 0x40000000]
[    7.092761] pci 0002:00:00.0: BAR 6: assigned [mem 0xf0200000-0xf020ffff pref]
[    7.093423] pci 0002:00:00.0: PCI bridge to [bus 01-ff]
[    7.096789] pcieport 0002:00:00.0: PME: Signaling with IRQ 43
[    7.097857] pcieport 0002:00:00.0: AER: enabled with IRQ 43
[   23.016266] platform 3c0000000.pcie: deferred probe pending

And thank you for just taking a look at it. I really appreciate it!

[inindev]

are you hitting this?
#36

broken 6.1.0-31-arm64
confirmed resolved: 6.1.0-32-arm64

apt update to take the latest kernel

[eggcllnt]

I also have log from successfully loading nvme. It happens randomly, no particular sequences

dmesg | grep -i -E 'pci|nvme'
[    0.768272] PCI: CLS 0 bytes, default 64
[    0.847158] rockchip-dw-pcie 3c0000000.pcie: host bridge /pcie@fe260000 ranges:
[    0.847253] rockchip-dw-pcie 3c0000000.pcie:       IO 0x00f4100000..0x00f41fffff -> 0x00f4100000
[    0.847306] rockchip-dw-pcie 3c0000000.pcie:      MEM 0x00f4200000..0x00f5ffffff -> 0x00f4200000
[    0.847345] rockchip-dw-pcie 3c0000000.pcie:      MEM 0x0300000000..0x033fffffff -> 0x0040000000
[    0.847641] rockchip-dw-pcie 3c0000000.pcie: iATU: unroll T, 8 ob, 8 ib, align 64K, limit 8G
[    1.054951] rockchip-dw-pcie 3c0000000.pcie: PCIe Gen.2 x1 link up
[    1.055700] rockchip-dw-pcie 3c0000000.pcie: PCI host bridge to bus 0000:00
[    1.055756] pci_bus 0000:00: root bus resource [bus 00-0f]
[    1.055788] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff] (bus address [0xf4100000-0xf41fffff])
[    1.055819] pci_bus 0000:00: root bus resource [mem 0xf4200000-0xf5ffffff]
[    1.055846] pci_bus 0000:00: root bus resource [mem 0x300000000-0x33fffffff] (bus address [0x40000000-0x7fffffff])
[    1.055917] pci 0000:00:00.0: [1d87:3566] type 01 class 0x060400
[    1.055968] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    1.056071] pci 0000:00:00.0: supports D1 D2
[    1.056094] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    1.066535] pci_bus 0000:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
[    1.066723] pci 0000:01:00.0: [10ec:8125] type 00 class 0x020000
[    1.066873] pci 0000:01:00.0: reg 0x10: initial BAR value 0x00000000 invalid
[    1.066904] pci 0000:01:00.0: reg 0x10: [io  size 0x0100]
[    1.067015] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x0000ffff 64bit]
[    1.067097] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit]
[    1.067636] pci 0000:01:00.0: supports D1 D2
[    1.067661] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    1.079061] pci 0000:00:00.0: BAR 14: assigned [mem 0xf4200000-0xf42fffff]
[    1.079129] pci 0000:00:00.0: BAR 6: assigned [mem 0xf4300000-0xf430ffff pref]
[    1.079164] pci 0000:00:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    1.079198] pci 0000:01:00.0: BAR 2: assigned [mem 0xf4200000-0xf420ffff 64bit]
[    1.079273] pci 0000:01:00.0: BAR 4: assigned [mem 0xf4210000-0xf4213fff 64bit]
[    1.079346] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    1.079387] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    1.079411] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    1.079436] pci 0000:00:00.0:   bridge window [mem 0xf4200000-0xf42fffff]
[    1.082420] pcieport 0000:00:00.0: PME: Signaling with IRQ 22
[    1.083075] pcieport 0000:00:00.0: AER: enabled with IRQ 22
[    2.227215] rockchip-dw-pcie 3c0400000.pcie: host bridge /pcie@fe270000 ranges:
[    2.227285] rockchip-dw-pcie 3c0400000.pcie:       IO 0x00f2100000..0x00f21fffff -> 0x00f2100000
[    2.227335] rockchip-dw-pcie 3c0400000.pcie:      MEM 0x00f2200000..0x00f3ffffff -> 0x00f2200000
[    2.227392] rockchip-dw-pcie 3c0400000.pcie:      MEM 0x0340000000..0x037fffffff -> 0x0040000000
[    2.239044] rockchip-dw-pcie 3c0400000.pcie: iATU: unroll T, 8 ob, 8 ib, align 64K, limit 8G
[    2.446840] rockchip-dw-pcie 3c0400000.pcie: PCIe Gen.2 x1 link up
[    2.447074] rockchip-dw-pcie 3c0400000.pcie: PCI host bridge to bus 0001:00
[    2.447111] pci_bus 0001:00: root bus resource [bus 00-0f]
[    2.447141] pci_bus 0001:00: root bus resource [io  0x100000-0x1fffff] (bus address [0xf2100000-0xf21fffff])
[    2.447172] pci_bus 0001:00: root bus resource [mem 0xf2200000-0xf3ffffff]
[    2.447201] pci_bus 0001:00: root bus resource [mem 0x340000000-0x37fffffff] (bus address [0x40000000-0x7fffffff])
[    2.447275] pci 0001:00:00.0: [1d87:3566] type 01 class 0x060400
[    2.447327] pci 0001:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    2.447430] pci 0001:00:00.0: supports D1 D2
[    2.447454] pci 0001:00:00.0: PME# supported from D0 D1 D3hot
[    2.456327] pci_bus 0001:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
[    2.456515] pci 0001:01:00.0: [10ec:8125] type 00 class 0x020000
[    2.456623] pci 0001:01:00.0: reg 0x10: [io  0x0000-0x00ff]
[    2.456733] pci 0001:01:00.0: reg 0x18: [mem 0x00000000-0x0000ffff 64bit]
[    2.456816] pci 0001:01:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit]
[    2.457363] pci 0001:01:00.0: supports D1 D2
[    2.457388] pci 0001:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    2.466984] pci 0001:00:00.0: BAR 14: assigned [mem 0xf2200000-0xf22fffff]
[    2.467026] pci 0001:00:00.0: BAR 6: assigned [mem 0xf2300000-0xf230ffff pref]
[    2.467060] pci 0001:00:00.0: BAR 13: assigned [io  0x100000-0x100fff]
[    2.467094] pci 0001:01:00.0: BAR 2: assigned [mem 0xf2200000-0xf220ffff 64bit]
[    2.467171] pci 0001:01:00.0: BAR 4: assigned [mem 0xf2210000-0xf2213fff 64bit]
[    2.467245] pci 0001:01:00.0: BAR 0: assigned [io  0x100000-0x1000ff]
[    2.467286] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    2.467311] pci 0001:00:00.0:   bridge window [io  0x100000-0x100fff]
[    2.467338] pci 0001:00:00.0:   bridge window [mem 0xf2200000-0xf22fffff]
[    2.470440] pcieport 0001:00:00.0: PME: Signaling with IRQ 73
[    2.471031] pcieport 0001:00:00.0: AER: enabled with IRQ 73
[    2.472331] rockchip-dw-pcie 3c0800000.pcie: host bridge /pcie@fe280000 ranges:
[    2.472408] rockchip-dw-pcie 3c0800000.pcie:       IO 0x00f0100000..0x00f01fffff -> 0x00f0100000
[    2.472461] rockchip-dw-pcie 3c0800000.pcie:      MEM 0x00f0200000..0x00f1ffffff -> 0x00f0200000
[    2.472503] rockchip-dw-pcie 3c0800000.pcie:      MEM 0x0380000000..0x03bfffffff -> 0x0040000000
[    2.484396] rockchip-dw-pcie 3c0800000.pcie: iATU: unroll T, 8 ob, 8 ib, align 64K, limit 8G
[    2.690833] rockchip-dw-pcie 3c0800000.pcie: PCIe Gen.1 x1 link up
[    2.691043] rockchip-dw-pcie 3c0800000.pcie: PCI host bridge to bus 0002:00
[    2.691078] pci_bus 0002:00: root bus resource [bus 00-0f]
[    2.691109] pci_bus 0002:00: root bus resource [io  0x200000-0x2fffff] (bus address [0xf0100000-0xf01fffff])
[    2.691141] pci_bus 0002:00: root bus resource [mem 0xf0200000-0xf1ffffff]
[    2.691169] pci_bus 0002:00: root bus resource [mem 0x380000000-0x3bfffffff] (bus address [0x40000000-0x7fffffff])
[    2.691240] pci 0002:00:00.0: [1d87:3566] type 01 class 0x060400
[    2.691283] pci 0002:00:00.0: reg 0x10: [mem 0x00000000-0x3fffffff]
[    2.691313] pci 0002:00:00.0: reg 0x14: [mem 0x00000000-0x3fffffff]
[    2.691344] pci 0002:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    2.691458] pci 0002:00:00.0: supports D1 D2
[    2.691482] pci 0002:00:00.0: PME# supported from D0 D1 D3hot
[    2.700035] pci 0002:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.700395] pci 0002:01:00.0: [1344:5416] type 00 class 0x010802
[    2.700560] pci 0002:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[    2.701717] pci 0002:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0002:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[    2.737320] pci_bus 0002:01: busn_res: [bus 01-0f] end is updated to 01
[    2.737388] pci 0002:00:00.0: BAR 0: assigned [mem 0x380000000-0x3bfffffff]
[    2.737425] pci 0002:00:00.0: BAR 1: no space for [mem size 0x40000000]
[    2.737453] pci 0002:00:00.0: BAR 1: failed to assign [mem size 0x40000000]
[    2.737479] pci 0002:00:00.0: BAR 14: assigned [mem 0xf0200000-0xf02fffff]
[    2.737507] pci 0002:00:00.0: BAR 6: assigned [mem 0xf0300000-0xf030ffff pref]
[    2.737543] pci 0002:01:00.0: BAR 0: assigned [mem 0xf0200000-0xf0203fff 64bit]
[    2.737613] pci 0002:00:00.0: PCI bridge to [bus 01]
[    2.737640] pci 0002:00:00.0:   bridge window [mem 0xf0200000-0xf02fffff]
[    2.738084] pcieport 0002:00:00.0: enabling device (0000 -> 0002)
[    2.740625] pcieport 0002:00:00.0: PME: Signaling with IRQ 75
[    2.741204] pcieport 0002:00:00.0: AER: enabled with IRQ 75
[    2.741917] nvme nvme0: pci function 0002:01:00.0
[    2.741967] nvme 0002:01:00.0: enabling device (0000 -> 0002)
[    2.810027] nvme nvme0: allocated 64 MiB host memory buffer.
[    2.856851] nvme nvme0: 4/0/0 default/read/poll queues
[    2.862622]  nvme0n1: p1

[eggcllnt]

I did an upgrade, now I'm on 6.1.0-34, but the issue the same. I saw there was confirmed resolved: 6.1.0-32-arm64, but no luck for me.

uname -a
Linux bookworm-nanopi-r5s 6.1.0-34-arm64 #1 SMP Debian 6.1.135-1 (2025-04-25) aarch64 GNU/Linux

[inindev]

is the 6.12 line of kernels an option?

#36 (comment)

[eggcllnt]

uname -a
Linux bookworm-nanopi-r5s 6.12.25-arm64 #1 SMP Debian 6.12.25-1 (2025-04-25) aarch64 GNU/Linux

so, I have some progress. it's still not showing up, but at least I saw next, after updating to 6.12.25

sudo dmesg | grep -i -E 'pci|nvme'
...
[    7.623071] pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc
...

so, I've tried modify /boot/mk_extlinux by setting EXLT_CMD_LINE='rootwait pci=realloc'. after reboot I can see it is enabled, but lsblk still do not show anything

[    6.665199] pci_bus 0002:00: root bus resource [bus 00-0f]
[    6.665714] pci_bus 0002:00: root bus resource [io  0x200000-0x2fffff] (bus address [0xf0100000-0xf01fffff])
[    6.666600] pci_bus 0002:00: root bus resource [mem 0xf0200000-0xf1ffffff]
[    6.667230] pci_bus 0002:00: root bus resource [mem 0x380000000-0x3bfffffff] (bus address [0x40000000-0x7fffffff])
[    6.668273] pci 0002:00:00.0: [1d87:3566] type 01 class 0x060400 PCIe Root Port
[    6.668961] pci 0002:00:00.0: BAR 0 [mem 0x00000000-0x3fffffff]
[    6.669511] pci 0002:00:00.0: BAR 1 [mem 0x00000000-0x3fffffff]
[    6.670057] pci 0002:00:00.0: ROM [mem 0x00000000-0x0000ffff pref]
[    6.670623] pci 0002:00:00.0: PCI bridge to [bus 00]
[    6.671085] pci 0002:00:00.0:   bridge window [io  0x0000-0x0fff]
[    6.671669] pci 0002:00:00.0:   bridge window [mem 0x00000000-0x000fffff]
[    6.672297] pci 0002:00:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[    6.673090] pci 0002:00:00.0: supports D1 D2
[    6.673494] pci 0002:00:00.0: PME# supported from D0 D1 D3hot
[    6.679599] pci 0002:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    6.680678] pci 0002:01:00.0: [1344:5416] type 00 class 0x010802 PCIe Endpoint
[    6.681477] pci 0002:01:00.0: BAR 0 [mem 0x00000000-0x00003fff 64bit]
[    6.683402] pci 0002:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0002:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[    6.739602] pci 0002:01:00.0: calc_l12_pwron: Invalid T_PwrOn scale: 3
[    6.740308] pci_bus 0002:01: busn_res: [bus 01-0f] end is updated to 01
[    6.740934] pci_bus 0002:00: max bus depth: 1 pci_try_num: 2
[    6.741487] pci 0002:00:00.0: BAR 0 [mem 0x380000000-0x3bfffffff]: assigned
[    6.742131] pci 0002:00:00.0: BAR 1 [mem size 0x40000000]: can't assign; no space
[    6.742813] pci 0002:00:00.0: BAR 1 [mem size 0x40000000]: failed to assign
[    6.743451] pci 0002:00:00.0: bridge window [mem 0xf0200000-0xf02fffff]: assigned
[    6.744194] pci 0002:00:00.0: ROM [mem 0xf0300000-0xf030ffff pref]: assigned
[    6.744860] pci 0002:01:00.0: BAR 0 [mem 0xf0200000-0xf0203fff 64bit]: assigned
[    6.745537] pci 0002:01:00.0: BAR 0: error updating (0xf0200004 != 0xffffffff)
[    6.746197] pci 0002:01:00.0: BAR 0: error updating (high 0x00000000 != 0xffffffff)
[    6.746894] pci 0002:00:00.0: PCI bridge to [bus 01]
[    6.747355] pci 0002:00:00.0:   bridge window [mem 0xf0200000-0xf02fffff]
[    6.748006] pci_bus 0002:00: No. 2 try to assign unassigned res
[    6.748567] pci 0002:00:00.0: BAR 1 [mem size 0x40000000]: can't assign; no space
[    6.749250] pci 0002:00:00.0: BAR 1 [mem size 0x40000000]: failed to assign
[    6.749887] pci 0002:00:00.0: PCI bridge to [bus 01]
[    6.750350] pci 0002:00:00.0:   bridge window [mem 0xf0200000-0xf02fffff]
[    6.750974] pci_bus 0002:00: resource 4 [io  0x200000-0x2fffff]
[    6.751542] pci_bus 0002:00: resource 5 [mem 0xf0200000-0xf1ffffff]
[    6.752117] pci_bus 0002:00: resource 6 [mem 0x380000000-0x3bfffffff]
[    6.752704] pci_bus 0002:01: resource 1 [mem 0xf0200000-0xf02fffff]
[    6.753620] pcieport 0002:00:00.0: enabling device (0000 -> 0002)
[    6.757198] pcieport 0002:00:00.0: PME: Signaling with IRQ 72
[    6.758647] pcieport 0002:00:00.0: AER: enabled with IRQ 72
[   12.990619] nvme nvme0: pci function 0002:01:00.0
[   12.991119] nvme 0002:01:00.0: enabling device (0000 -> 0002)

lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
mmcblk1      179:0    0 28.9G  0 disk
└─mmcblk1p1  179:1    0 28.9G  0 part /
mmcblk1boot0 179:256  0    4M  1 disk
mmcblk1boot1 179:512  0    4M  1 disk

lspci -vv
0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 65
        Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
        I/O behind bridge: 1000-1fff [size=4K] [16-bit]
        Memory behind bridge: f4200000-f42fffff [size=1M] [32-bit]
        Prefetchable memory behind bridge: [disabled] [64-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        Expansion ROM at f4300000 [virtual] [disabled] [size=64K]
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: <access denied>
        Kernel driver in use: pcieport

0000:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 65
        Region 0: I/O ports at 1000 [size=256]
        Region 2: Memory at f4200000 (64-bit, non-prefetchable) [size=64K]
        Region 4: Memory at f4210000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: r8169
        Kernel modules: r8169

0001:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 69
        Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
        I/O behind bridge: 100000-100fff [size=4K] [16-bit]
        Memory behind bridge: f2200000-f22fffff [size=1M] [32-bit]
        Prefetchable memory behind bridge: [disabled] [64-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        Expansion ROM at f2300000 [virtual] [disabled] [size=64K]
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: <access denied>
        Kernel driver in use: pcieport

0001:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 69
        Region 0: I/O ports at 100000 [size=256]
        Region 2: Memory at f2200000 (64-bit, non-prefetchable) [size=64K]
        Region 4: Memory at f2210000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: r8169
        Kernel modules: r8169

0002:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01) (prog-if 00 [Normal decode])
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 72
        Region 0: Memory at 380000000 (32-bit, non-prefetchable) [size=1G]
        Bus: primary=00, secondary=00, subordinate=01, sec-latency=0
        I/O behind bridge: [disabled] [16-bit]
        Memory behind bridge: f0200000-f02fffff [size=1M] [32-bit]
        Prefetchable memory behind bridge: [disabled] [64-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        Expansion ROM at f0300000 [virtual] [disabled] [size=64K]
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: <access denied>
        Kernel driver in use: pcieport

0002:01:00.0 Non-Volatile memory controller: Micron Technology Inc 2550 NVMe SSD (DRAM-less) (rev 01) (prog-if 02 [NVM Express])
        Subsystem: Micron Technology Inc 2550 NVMe SSD (DRAM-less)
        Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 72
        Region 0: Memory at f0200000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel modules: nvme

[inindev]

The 6.1 debian kernels did not ship with a rk3568-nanopi-r5s.dtb device tree. The 6.12 kernels do, we should switch to using it. Yours may not be symbolic links as about a year ago I adjusted /etc/kernel/postinst.d/dtb_cp to prefer device trees shipped with the kernel.

$ ls -al /boot/rk*
-rw-r--r-- 1 root root 58986 Jul  2  2024 /boot/rk3568-nanopi-r5s.dtb
lrwxrwxrwx 1 root root    21 Jul  2  2024 /boot/rk3568-nanopi-r5s.dtb-6.1.0-22-arm64 -> rk3568-nanopi-r5s.dtb
lrwxrwxrwx 1 root root    21 Feb 15 15:32 /boot/rk3568-nanopi-r5s.dtb-6.12.12-arm64 -> rk3568-nanopi-r5s.dtb
lrwxrwxrwx 1 root root    21 Mar 11 04:57 /boot/rk3568-nanopi-r5s.dtb-6.12.17-arm64 -> rk3568-nanopi-r5s.dtb
lrwxrwxrwx 1 root root    21 Apr  6 14:02 /boot/rk3568-nanopi-r5s.dtb-6.12.21-arm64 -> rk3568-nanopi-r5s.dtb
lrwxrwxrwx 1 root root    21 May  4 04:02 /boot/rk3568-nanopi-r5s.dtb-6.12.22-arm64 -> rk3568-nanopi-r5s.dtb

Here is the one we want:

$ ls -al /lib/linux-image-6.12.25-arm64/rockchip/rk3568-nanopi-r5s.dtb
-rw-r--r-- 1 root root 61027 Apr 25 11:07 /lib/linux-image-6.12.25-arm64/rockchip/rk3568-nanopi-r5s.dtb

Use it instead:

$ sudo rm /boot/rk3568-nanopi-r5s.dtb-6.12.25-arm64
$ sudo cp /lib/linux-image-6.12.25-arm64/rockchip/rk3568-nanopi-r5s.dtb /boot/rk3568-nanopi-r5s.dtb-6.12.25-arm64

I am unsure if this is the issue, but its a good first place to look as debian 6.12.25 is a known working kernel.

[eggcllnt]

Unfortunately the same behaviour. Nothing changed.
Is there some steps/commands/logs that I can take a look to identify more information?
Thanks!

[MichaIng]

Linux 6.16 (maybe earlier) has one part of those "errors" (well, logged as info severity only) solved: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit?id=fbea35a

So

busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])

does not show up anymore.

We have also various reports for different RK356x SoC boards among our users with failing PCIe device detection, also embedded ones like 2.5 Gbit Ethernet ports attached via PCIe. And it indeed happens randomly:

• Some individual boards are affected more, others with same board revision are not or less.
• Sometimes all PCIe devices are detected, sometimes one is missing, sometimes another one.

Above commit sadly did not solve this, so I think the suspicious kernel messages are unrelated. Based on the output it really looks like there was just no device attached to the "port", whether it is an actual port like M.2, or just internally wired in the PCB.

While several RK3568 and RK3566 boards are affected, like NanoPi R5S/R5C and Orange Pi 3B, with multiple reports each over at least the last year with multiple mainline Linux versions, we have no such for any RK3588 or other SoCs. So to me it looks like this is caused by some RK356x specific parts of the kernel (or hardware/chip design). Both share the same rk3568.dtsi, so it is not surprising.

I have no idea how to further debug, found this matching issue here randomly. Not the same project, but maybe we can share findings and ideas how to get to the bottom of this. I will try to get some attention on the Linux mailing list as well.

[inindev]

It was not a big deal, there were two Debian kernel builds which had the defect before the change was reverted. Working / current images can be found here: https://github.com/inindev/debian-image

[MichaIng]

Interesting. Do you know the particular commit or kernel config change done by Debian? Might be well some issue with the build/config instead of the sources then.

[inindev]

It looks like it impacted these three Debian kernels:
6.1.0-29-arm64
6.1.0-30-arm64
6.1.0-31-arm64
#36

This same defect was checked into 6.12.25 and 6.14.4 at the time but these were not part of Bookworm stable / Debian.