Bug:IPinIP Hash Test Fails due to Incorrect Outer IP Protocol

[aeedara-nokia]

Is it platform specific

generic

Importance or Severity

Critical

Description of the bug

The IPinIPHashTest in T2 topology fails because packet sent is sent out wrong with Outer IP Protocol field set to 6 (TCP)
instead of the required 4 (IP-in-IP Encapsulation) and with wrong dst mac and src mac. This causes the DUT to drop the packet, resulting in two failures related to IPinIphash.

PR #17503 made OOP changes to some of the code used here (hash_test.py) to generate the packet before this PR tests passed.

Steps to Reproduce

Run test_fib.py from SONiC mgmt on T2 topology.

Actual Behavior and Expected Behavior

Actual Behavior
The test fails with an assertion error (AssertionError: Did not receive expected packet...).
The debug logs confirm the packet being sent has an incorrect outer protocol, proving the packet is malformed for a tunnel:

The test should pass, and the debug logs should confirm that the packet sent has the correct IP-in-IP encapsulation protocol:

Relevant log output

Not working case with commit #17503:
06:39:56.764  root      : INFO    : ++++++++ Sun Oct 12 06:39:56 2025 ++++++++
06:39:56.953  root      : INFO    : ROCEv2 support found in Scapy
06:39:56.954  root      : INFO    : ERSPAN support found in Scapy
06:39:56.954  root      : INFO    : GENEVE support found in Scapy
06:39:56.955  root      : INFO    : MPLS support found in Scapy
06:39:56.956  root      : INFO    : IGMP support found in Scapy
06:39:57.181  root      : INFO    : Importing platform: remote
06:39:57.182  root      : INFO    : port map: {(0, 0): 'eth0', (0, 1): 'eth1', (0, 2): 'eth2', (0, 3): 'eth3', (0, 4): 'eth4', (0, 5): 'eth5', (0, 6): 'eth6', (0, 7): 'eth7', (0, 8): 'eth8', (0, 9): 'eth9', (0, 10): 'eth10', (0, 11): 'eth11', (0, 12): 'eth12', (0, 13): 'eth13', (0, 14): 'eth14', (0, 15): 'eth15', (0, 16): 'eth16', (0, 17): 'eth17', (0, 18): 'eth18', (0, 19): 'eth19', (0, 20): 'eth20', (0, 21): 'eth21', (0, 22): 'eth22', (0, 23): 'eth23', (0, 24): 'eth24', (0, 25): 'eth25', (0, 26): 'eth26', (0, 27): 'eth27', (0, 28): 'eth28', (0, 29): 'eth29', (0, 30): 'eth30', (0, 31): 'eth31', (0, 32): 'eth32', (0, 33): 'eth33', (0, 34): 'eth34', (0, 35): 'eth35', (0, 36): 'eth36', (0, 37): 'eth37', (0, 38): 'eth38', (0, 39): 'eth39', (0, 40): 'eth40', (0, 41): 'eth41', (0, 42): 'eth42', (0, 43): 'eth43', (0, 44): 'eth44', (0, 45): 'eth45', (0, 46): 'eth46', (0, 47): 'eth47', (0, 48): 'eth48', (0, 49): 'eth49', (0, 50): 'eth50', (0, 51): 'eth51', (0, 52): 'eth52', (0, 53): 'eth53', (0, 54): 'eth54', (0, 55): 'eth55', (0, 56): 'eth56', (0, 57): 'eth57', (0, 58): 'eth58', (0, 59): 'eth59', (0, 60): 'eth60', (0, 61): 'eth61', (0, 62): 'eth62', (0, 63): 'eth63', (0, 64): 'eth64', (0, 65): 'eth65', (0, 66): 'eth66', (0, 67): 'eth67', (0, 68): 'eth68', (0, 69): 'eth69', (0, 70): 'eth70', (0, 71): 'eth71'}
06:39:57.182  root      : INFO    : Autogen random seed: 43879429
06:39:57.226  root      : INFO    : *** TEST RUN START: Sun Oct 12 06:39:57 2025
06:40:03.942  root      : INFO    : List of hash_keys: ['inner_length']
06:40:03.942  root      : INFO    : hash test hash_key: inner_length
06:40:03.942  root      : INFO    : outer_src_ip=80.1.0.31, outer_dst_ip=80.1.0.32, src_port=56, exp_port_lists=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35]]
06:40:03.942  root      : INFO    : Checking hash key inner_length, exp_ports=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35]], outer_src_ip=80.1.0.31, outer_dst_ip=80.1.0.32
06:40:03.969  root      : INFO    : Sent Ether(src=00:06:07:08:09:0a, dst=00:01:02:03:04:05)/IP(src=8.0.0.0, dst=9.0.0.0, proto=6)/IP(src=8.0.0.0, dst=9.0.0.0, proto=6)/TCP(sport=1234, dport=80 on port 56)
06:40:03.969  dataplane : WARNING : Dataplane poll with exp_pkt but no port number
06:40:04.989  root      : INFO    : ** END TEST CASE hash_test.IPinIPHashTest
06:40:04.990  root      : INFO    : *** TEST RUN END  : Sun Oct 12 06:40:04 2025
06:40:04.990  dataplane : INFO    : Thread exit


working prior to the commit #17503:
---------------------------------------------------------------------------------------------------------------------------------------------
17:14:40.191  root      : INFO    : Sent Ether(src=4e:ba:d7:b2:c0:1d, dst=00:31:26:7e:f1:38)/IP(src=80.1.0.31, dst=80.1.0.32, proto=4)/IP(src=8.0.0.0, dst=9.0.0.0, proto=6)/TCP(sport=1234, dport=80 on port 13)
17:14:40.191  dataplane : WARNING : Dataplane poll with exp_pkt but no port number
17:14:40.195  root      : INFO    : Received packet at 24
17:14:40.216  root      : INFO    : Checking hash key inner_length, exp_ports=[[0, 1, 2, 3, 4, 5, 6, 7, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]], outer_src_ip=80.1.0.31, outer_dst_ip=80.1.0.32
17:14:40.240  root      : INFO    : Sent Ether(src=4e:ba:d7:b2:c0:1d, dst=00:31:26:7e:f1:38)/IP(src=80.1.0.31, dst=80.1.0.32, proto=4)/IP(src=8.0.0.0, dst=9.0.0.0, proto=6)/TCP(sport=1234, dport=80 on port 13)
17:14:40.240  dataplane : WARNING : Dataplane poll with exp_pkt but no port number
17:14:40.241  root      : INFO    : Received packet at 24
17:14:40.261  root      : INFO    : Checking hash key inner_length, exp_ports=[[0, 1, 2, 3, 4, 5, 6, 7, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]], outer_src_ip=80.1.0.31, outer_dst_ip=80.1.0.32
17:14:40.285  root      : INFO    : Sent Ether(src=4e:ba:d7:b2:c0:1d, dst=00:31:26:7e:f1:38)/IP(src=80.1.0.31, dst=80.1.0.32, proto=4)/IP(src=8.0.0.0, dst=9.0.0.0, proto=6)/TCP(sport=1234, dport=80 on port 13)
17:14:40.285  dataplane : WARNING : Dataplane poll with exp_pkt but no port number
17:14:40.286  root      : INFO    : Received packet at 24
17:14:40.306  root      : INFO    : Checking hash key inner_length, exp_ports=[[0, 1, 2, 3, 4, 5, 6, 7, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]], outer_src_ip=80.1.0.31, outer_dst_ip=80.1.0.32
17:14:40.330  root      : INFO    : Sent Ether(src=4e:ba:d7:b2:c0:1d, dst=00:31:26:7e:f1:38)/IP(src=80.1.0.31, dst=80.1.0.32, proto=4)/IP(src=8.0.0.0, dst=9.0.0.0, proto=6)/TCP(sport=1234, dport=80 on port 13)
17:14:40.330  dataplane : WARNING : Dataplane poll with exp_pkt but no port number
17:14:40.331  root      : INFO    : Received packet at 24
17:14:40.351  root      : INFO    : Checking hash key inner_length, exp_ports=[[0, 1, 2, 3, 4, 5, 6, 7, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]], outer_src_ip=80.1.0.31, outer_dst_ip=80.1.0.32
17:14:40.376  root      : INFO    : Sent Ether(src=4e:ba:d7:b2:c0:1d, dst=00:31:26:7e:f1:38)/IP(src=80.1.0.31, dst=80.1.0.32, proto=4)/IP(src=8.0.0.0, dst=9.0.0.0, proto=6)/TCP(sport=1234, dport=80 on port 13)
17:14:40.376  dataplane : WARNING : Dataplane poll with exp_pkt but no port number
17:14:40.377  root      : INFO    : Received packet at 24
17:14:40.397  root      : INFO    : Checking hash key inner_length, exp_ports=[[0, 1, 2, 3, 4, 5, 6, 7, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]], outer_src_ip=80.1.0.31, outer_dst_ip=80.1.0.32
17:14:40.440  root      : INFO    : Sent Ether(src=4e:ba:d7:b2:c0:1d, dst=00:31:26:7e:f1:38)/IP(src=80.1.0.31, dst=80.1.0.32, proto=4)/IP(src=8.0.0.0, dst=9.0.0.0, proto=6)/TCP(sport=1234, dport=80 on port 13)
17:14:40.440  dataplane : WARNING : Dataplane poll with exp_pkt but no port number
17:14:40.442  root      : INFO    : Received packet at 24
17:14:40.462  root      : INFO    : Checking hash key inner_length, exp_ports=[[0, 1, 2, 3, 4, 5, 6, 7, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]], outer_src_ip=80.1.0.31, outer_dst_ip=80.1.0.32
17:14:40.511  root      : INFO    : Sent Ether(src=4e:ba:d7:b2:c0:1d, dst=00:31:26:7e:f1:38)/IP(src=80.1.0.31, dst=80.1.0.32, proto=4)/IP(src=8.0.0.0, dst=9.0.0.0, proto=6)/TCP(sport=1234, dport=80 on port 13)
17:14:40.512  dataplane : WARNING : Dataplane poll with exp_pkt but no port number
17:14:40.513  root      : INFO    : Received packet at 24
17:14:40.534  root      : INFO    : Checking hash key inner_length, exp_ports=[[0, 1, 2, 3, 4, 5, 6, 7, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]], outer_src_ip=80.1.0.31, outer_dst_ip=80.1.0.32
17:14:40.583  root      : INFO    : Sent Ether(src=4e:ba:d7:b2:c0:1d, dst=00:31:26:7e:f1:38)/IP(src=80.1.0.31, dst=80.1.0.32, proto=4)/IP(src=8.0.0.0, dst=9.0.0.0, proto=6)/TCP(sport=1234, dport=80 on port 13)
17:14:40.583  dataplane : WARNING : Dataplane poll with exp_pkt but no port number
17:14:40.585  root      : INFO    : Received packet at 24
17:14:40.605  root      : INFO    : Checking hash key inner_length, exp_ports=[[0, 1, 2, 3, 4, 5, 6, 7, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]], outer_src_ip=80.1.0.31, outer_dst_ip=80.1.0.32
17:14:40.654  root      : INFO    : Sent Ether(src=4e:ba:d7:b2:c0:1d, dst=00:31:26:7e:f1:38)/IP(src=80.1.0.31, dst=80.1.0.32, proto=4)/IP(src=8.0.0.0, dst=9.0.0.0, proto=6)/TCP(sport=1234, dport=80 on port 13)
17:14:40.654  dataplane : WARNING : Dataplane poll with exp_pkt but no port number
17:14:40.656  root      : INFO    : Received packet at 24
17:14:40.677  root      : INFO    : Checking hash key inner_length, exp_ports=[[0, 1, 2, 3, 4, 5, 6, 7, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]], outer_src_ip=80.1.0.31, outer_dst_ip=80.1.0.32
17:14:40.725  root      : INFO    : Sent Ether(src=4e:ba:d7:b2:c0:1d, dst=00:31:26:7e:f1:38)/IP(src=80.1.0.31, dst=80.1.0.32, proto=4)/IP(src=8.0.0.0, dst=9.0.0.0, proto=6)/TCP(sport=1234, dport=80 on port 13)
17:14:40.726  dataplane : WARNING : Dataplane poll with exp_pkt but no port number
17:14:40.727  root      : INFO    : Received packet at 24

Output of show version

admin@ixre-egl-board202:~$ show version

SONiC Software Version: SONiC.HEAD.1189706-nokia-master-4b078640b
SONiC OS Version: 12
Distribution: Debian 12.12
Kernel: 6.1.0-29-2-amd64
Build commit: 4b078640b
Build date: Fri Oct 10 17:06:24 UTC 2025
Built by: gitlab-runner@sonic-build-server03

Attach files (if any)

No response

[aeedara-nokia]

@gshemesh2 could you please check we are seeing this issue after this commit #17503, are there any other changes we'd need , we pulled in changes from #20881 this also but no luck.

[aeedara-nokia]

Identified the root cause of the issue , testing of the fix in progress

[rlhui]

@roy-sror , please note

[roy-sror]

@rlhui , @aeedara-nokia - thanks for letting us know. I believe the main reason it was not found by internal testing is that we don't have any T2 testbed to test this change.

[abdosi]

@bingwang-ms : can you check on this regression with Nvidia.