loxilb standalone nat has effective MTU 1170

[shcherbak]

descriptionr
Loxilb in standalone mode (onearm) cuts mtu 1500 to effective size 1170 and produces instant retransmittions, package drops , unavailable to work with big packages

to reproduce

1. install loxilb in standalone onearm mode according to documentation (172.29.0.56 in examples)
2. ping -M do -s 1472 -c 5 172.29.0.56 -- 100% drops
   PING 172.29.0.56 (172.29.0.56) 1472(1500) bytes of data.
3. ping -M do -s 1162 -c 5 172.29.0.56 -- is ok

Expected behavior
response on ping of 1472 bytes

Screenshots
here is screenshot of l2 tcpdump with retransmittions mess

here is screenshot with tcpdump of losted pings (no respose found)

Environment (please complete the following information):

• OS: debian 12, 13
• Kernel Version: 6.16.3+deb13-amd64 and 6.1.0-39-amd64
• LoxiLB Version: 0.9.8-beta 2025_01_27_12h:43m-
• Cloud Environment: On-prem
• K8s Version -

[backguynn]

Hello, could you please share the output of the loxicmd get lb -o wide command?

[backguynn]

I have looked into loxilb, but I couldn't find any evidence that it restricts the MTU. Loxilb does not internally limit the MTU size; it simply uses the MTU of the interface provided by the host or container as is.

Could you please share the actual MTU settings of the host and the container where loxilb is currently running?

Check the host's actual interface MTU

ip link show

Check the Docker network MTU (If using Docker)

docker network inspect bridge

Check the interface MTU inside the loxilb container

docker exec loxilb ip link show

[shcherbak]

this is stand-alone setup, dedicated loxilb host, no docker
as You can see all interfaces has 1500, but i have persistent reproduce that no pass for not fragmented ping larger then 1170

ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:b7:f0:68:09:d3 brd ff:ff:ff:ff:ff:ff
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:85:f6:0d:72:11 brd ff:ff:ff:ff:ff:ff
4: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:3d:23:59:a8:8d brd ff:ff:ff:ff:ff:ff
5: llb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdpgeneric qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether b6:e5:a8:18:24:eb brd ff:ff:ff:ff:ff:ff
    prog/xdp id 626 name xdp_packet_func tag bdb0fdac0d862b6c jited 

loxicmd get lb -o wide
|   EXT IP   | SEC IPS | SOURCES | HOST | PORT  | PROTO | NAME | MARK | SEL  |  MODE  |  ENDPOINT   | EPORT | WEIGHT | STATE |    COUNTERS    |
|------------|---------|---------|------|-------|-------|------|------|------|--------|-------------|-------|--------|-------|----------------|
| 172.29.0.3 |         |         |      |  8090 | tcp   |      |    0 | hash | onearm | 10.12.18.52 |  8090 |      2 | -     | 127:20751      |
| 172.29.0.3 |         |         |      |  8140 | tcp   |      |    0 | hash | onearm | 10.12.18.31 |  8140 |      1 | -     | 0:0            |
| 172.29.0.3 |         |         |      |  8428 | tcp   |      |    0 | hash | onearm | 10.12.18.26 |  8428 |      1 | -     | 0:0            |
| 172.29.0.3 |         |         |      |  9042 | tcp   |      |    0 | hash | onearm | 10.4.14.1   |  9042 |      1 | -     | 80721:13413736 |
| 172.29.0.3 |         |         |      | 24224 | tcp   |      |    0 | hash | onearm | 10.12.18.17 | 24224 |      2 | -     | 0:0            |
| 172.29.0.4 |         |         |      |   443 | tcp   |      |    0 | hash | onearm | 10.12.18.61 |   443 |      2 | -     | 0:0            |
| 172.29.0.4 |         |         |      | 24224 | tcp   |      |    0 | hash | onearm | 10.12.18.17 | 24224 |      2 | -     | 0:0            |

loxicmd get port -o wide
| INDEX | PORTNAME |        MAC        | LINK/STATE  | MTU  | ISACTIVE/BPF |           STATISTICS           |             L3INFO             |    L2INFO     | SYNC |
|       |          |                   |             |      |  PORT TYPE   |                                |                                |               |      |
|-------|----------|-------------------|-------------|------|--------------|--------------------------------|--------------------------------|---------------|------|
|     1 | enp1s0   | 02:b7:f0:68:09:d3 | false/false | 1500 | true/true    | rx/tx byte : 0/0               | Routed: false                  | IsPVID: true  |    0 |
|       |          |                   |             |      | phy          | rx/tx packets : 0/0            | IPv4 : []                      | VID : 3801    |      |
|       |          |                   |             |      |              | rx/tx error : 0/0              | IPv6 : []                      |               |      |
|     2 | vlan3801 | aa:bb:cc:dd:ee:ff | true/true   | 9000 | true/false   | rx/tx byte : 0/0               | Routed: false                  | IsPVID: false |    0 |
|       |          |                   |             |      | vlan         | rx/tx packets : 0/0            | IPv4 : []                      | VID : 3801    |      |
|       |          |                   |             |      |              | rx/tx error : 0/0              | IPv6 : []                      |               |      |
|     3 | enp2s0   | 02:85:f6:0d:72:11 | true/true   | 1500 | true/true    | rx/tx byte :                   | Routed: true IPv4 :            | IsPVID: true  |    0 |
|       |          |                   |             |      | phy          | 118500632205/23134676318       | [172.29.0.54/19 (P) ]  IPv6    | VID : 3803    |      |
|       |          |                   |             |      |              |  rx/tx packets :               | : [fe80::85:f6ff:fe0d:7211/64  |               |      |
|       |          |                   |             |      |              | 176929826/175400377  rx/tx     | (P) ]                          |               |      |
|       |          |                   |             |      |              | error : 0/0                    |                                |               |      |
|     4 | vlan3803 | aa:bb:cc:dd:ee:ff | true/true   | 9000 | true/false   | rx/tx byte : 0/0               | Routed: false                  | IsPVID: false |    0 |
|       |          |                   |             |      | vlan         | rx/tx packets : 0/0            | IPv4 : []                      | VID : 3803    |      |
|       |          |                   |             |      |              | rx/tx error : 0/0              | IPv6 : []                      |               |      |
|     5 | enp3s0   | 02:3d:23:59:a8:8d | true/true   | 1500 | true/true    | rx/tx byte :                   | Routed: true IPv4 :            | IsPVID: true  |    0 |
|       |          |                   |             |      | phy          | 42130461598/120419490839       | [10.4.4.1/11 (P) ]  IPv6 :     | VID : 3805    |      |
|       |          |                   |             |      |              |  rx/tx packets :               | [fe80::3d:23ff:fe59:a88d/64    |               |      |
|       |          |                   |             |      |              | 478639251/204439003  rx/tx     | (P) ]                          |               |      |
|       |          |                   |             |      |              | error : 0/0                    |                                |               |      |
|     6 | vlan3805 | aa:bb:cc:dd:ee:ff | true/true   | 9000 | true/false   | rx/tx byte : 0/0               | Routed: false                  | IsPVID: false |    0 |
|       |          |                   |             |      | vlan         | rx/tx packets : 0/0            | IPv4 : []                      | VID : 3805    |      |
|       |          |                   |             |      |              | rx/tx error : 0/0              | IPv6 : []                      |               |      |
|     7 | llb0     | b6:e5:a8:18:24:eb | true/true   | 1500 | true/true    | rx/tx byte : 0/0               | Routed: true IPv4 : []  IPv6   | IsPVID: true  |    0 |
|       |          |                   |             |      | phy          | rx/tx packets : 0/0            | : [fe80::900a:5eff:fe91:bfe/64 | VID : 3807    |      |
|       |          |                   |             |      |              | rx/tx error : 0/0              | (P) ]                          |               |      |
|     8 | vlan3807 | aa:bb:cc:dd:ee:ff | true/true   | 9000 | true/false   | rx/tx byte : 0/0               | Routed: false                  | IsPVID: false |    0 |
|       |          |                   |             |      | vlan         | rx/tx packets : 0/0            | IPv4 : []                      | VID : 3807    |      |
|       |          |                   |             |      |              | rx/tx error : 0/0              | IPv6 : []                      |               |      |

haw can i debug this behavior ?

[backguynn]

I have a few questions regarding your test:

1. Is the IP you are pinging the one configured as the external IP in the LB rule?

2. If so, is that IP assigned to the lo (loopback) interface or another interface? (While loxilb normally assigns the VIP to the lo interface, it does not do so in standalone - onearm mode.)

3. I didn't see anything related to ICMP in the LB rule you created. Therefore, your ping test was likely just checking whether the interface can receive unfragmented packets (rather than testing if NAT is working). In this case, loxilb does not touch or process those packets at all.

At this point, it is difficult to definitively conclude whether this is an issue with loxilb itself or an underlying environmental issue.
Therefore, we need to verify whether loxilb is the root cause, or if the issue stems from the path PMTU, NIC offloads, or the network driver.

Test A. Verify by eliminating loxilb's impact

Condition: Retest the same path after stopping loxilb or detaching XDP.
Execution: ping -M do -s 1200 [DST_IP]

If the exact same failure occurs even after removing loxilb, an environmental issue is the most likely cause.

Test B. Verify PMTU messages

Condition: Maintain the current environment.

Execution 1: tcpdump -ni [IFACE] icmp and 'icmp[0]=3 and icmp[1]=4'
Execution 2: ping -M do -s 1200 [DST_IP]

If "Fragmentation Needed" packets are observed, there is a high probability of a path PMTU issue.

Test C. Step-by-step verification of NIC offload impact

Condition: Same path, same packet size.

Execution 1: ethtool -K [IFACE] rx off tx off
Execution 2: ethtool -K [IFACE] gso off gro off tso off
Execution 3: ping -M do -s 1200 [DST_IP]

check in success/failure at each step.
If the symptom changes only at a specific step, there is a high probability of an interoperability issue with the NIC or its driver.

[backguynn]

And could you please let us know which documentation you referred to when configuring the onearm mode?
Also, if you could provide the steps to reproduce this issue, we will try to replicate it on our end.

[shcherbak]

Will prepare answer in next 2 days