chore: enable vrf converged topo

[auspham]

Description of PR

Enable the converged (multi-VRF) cEOS peer model across the virtual testbeds and add the framework changes needed so existing tests pass on every converged topology — one solution that fits all cEOS topologies, almost entirely without per-test-case edits (the framework adapts transparently; the only test touches are a single shared-helper call and one fix for a pre-existing bug).

In converged mode a single cEOS VM hosts every logical neighbor as a VRF under one BGP process, and each logical neighbor's interface is renamed on the shared VM. This breaks several assumptions baked into the framework: (1) EOS config writes target router bgp <asn> in the default VRF; (2) tests reference per-logical interface names from minigraph; (3) the legacy untagged backplane reachability (PTF bp ↔ cEOS ↔ DUT) is split into per-VRF VLAN sub-interfaces; (4) bash <cmd> invocations on the cEOS run in the default Linux netns, while all data routes live in per-VRF ns-<vrf> namespaces; and (5) tests that read the BGP table or push ad-hoc CLI assume the default VRF and global interface names.

Type of change

• Testbed and Framework (improvement)

Back port request

• 202311
• 202405
• 202411
• 202505
• 202511
• 202512
• 202605

Approach

Problem

1. EosHost.eos_config() calls land in the default VRF on converged VMs → BGP config writes (shutdown, no_shutdown_bgp, neighbor tweaks) silently miss.
2. EosHost.eos_command() / eos_config() calls reference per-logical interface names (e.g. Ethernet1) that don't exist on the shared converged VM.
3. PTF backplane interface has no IP on the parent device in converged mode, so untagged backplane traffic (used by e.g. BGP monitor / link-flap tests) has no return path.
4. ceos_topo_converger dropped disabled_host_interfaces from the converged topo, turning previously-admin-down ports into active ports and breaking buffer/QoS deployment.
5. bash <cmd> invocations on the cEOS (e.g. snmp/test_snmp_loopback.py) run in the route-less default Linux netns on converged, so they must be re-scoped into the per-VRF ns-<vrf> namespace that carries the route to the DUT.
6. Several per-role startup-config templates never grew a converged branch (t0-64-32-leaf, t2-leaf, t2-core, dpu-tor, t1-8-lag-spine, t1-8-lag-tor). On a converged topology they rendered only the prime's own config and silently dropped every merged sub-peer's interface IP and BGP neighbor — so the DUT's sessions to those sub-peers never established (pre-test BGP sanity failed), and the per-VRF Linux namespace ns-<vrf> was never created (so the netns-scoped snmpget failed with "Cannot open network namespace").
7. The converger selected one prime per (role, ASN), which exploded the prime count on real multi-ASN fabrics (e.g. t1-lag → 17, t1-lag-vpp → 17, t2 → 49). Each converged cEOS reserves significant memory, so this exhausted testbed RAM and failed add-topo.
8. Convergence also ran for non-cEOS neighbor deployments (SONiC-VS / cisco / csonic), which have no converged render path. Reshaping the topology there produced a DUT minigraph whose BGP neighbors the unconverged VS peers could not answer → "Not all bgp sessions are established".
9. add-topo failed with "Too many vlans. Maximum is 4" on converged topologies whose prime holds more than 4 front-panel links (t1-lag, t1-lag-vpp, t2). The converger surfaces the required front-panel count as max_fp_num_provided, but that override was only applied by the start vm_set action — while the cEOS front-panel bridges/veths are created by the add_topo action, which kept the default max_fp_num=4. Only 4 bridges were created and the subsequent bind failed.
10. deploy-mg failed with "IndexError: list index out of range" in topo_facts on multi-linecard t2 topologies. The converger dropped the topology-level dut_num, so minigraph generation defaulted it to 1 and overflowed the per-DUT interface_indexes list for sub-peers whose VM vlans carry dut_index > 0.
11. Raw BGP-table reads hit the wrong VRF. Tests that read the BGP table with a plain EosHost.run_command('show ip bgp …') — e.g. vxlan/test_vxlan_route_advertisement.py's scale check (show ip bgp community <c> | grep …) — read the prime's default VRF on a converged VM, which holds no logical-neighbor routes. The read returned empty, so the route check failed (and a retry path then surfaced a latent crash, see below).
12. Ad-hoc route advertisement landed in the wrong VRF and clobbered a loopback. Tests that advertise a network from a neighbor by pushing native CLI via run_command_list(["interface loopback N", …, "router bgp X", "network …"]) — e.g. vxlan/test_vnet_bgp_route_precedence.py — put the router bgp/network in the default VRF (the DUT never learned the route), and the lowercase interface loopback N idiom collided with the prime's real LoopbackN (interface names are global on EOS), corrupting another VRF's loopback.
13. exabgp port math used the wrong VM offset. The exabgp route-injector port is derived from a neighbor's VM offset; tests such as bgp/test_bgp_bbr.py computed it from the prime's offset instead of the merged neighbor's original offset, so they connected to the wrong exabgp instance.
14. A latent test bug crashed the scale retry path. vxlan/test_vxlan_route_advertisement.py's scale retry indexed a list with a string key (self.vxlan_test_setup['t2']['host'], where ['t2'] is a list) → TypeError. This crashes on any topology once the retry branch is reached; converged mode merely made the empty-read retry far more likely.

Solution

All framework changes are common-code and gated so non-converged / non-cEOS paths are byte-identical. The only test touches are a single shared-helper call (item 13) and a one-line fix for the pre-existing crash (item 14).

File	Change	Gate
tests/common/devices/eos.py	Wrap eos_config() / eos_command() to rewrite router bgp <asn> parents into router bgp <prime_asn> / vrf <vrf>, translate interface <name> tokens via intf_map, and rewrite bash <cmd> into bash sudo ip netns exec ns-<vrf> <cmd>. eos_command() also VRF-scopes raw show ip|ipv6 bgp reads (inject vrf <vrf> before any | grep, where EOS requires it) and ad-hoc router bgp / interface loopback config pushed via run_command_list — adding vrf <vrf> and renaming the throwaway interface loopback N to a collision-free Loopback<N+1000> in the VRF. VRF-aware get_route().	self.bgp_vrf / self.intf_map — both None on stock
tests/common/utilities.py	Add get_neighbor_exabgp_vm_offset(): map a converged neighbor back to its original VM offset (from multi_vrf_data) so exabgp-port math targets the correct instance.	returns the neighbor's unchanged offset on stock
tests/conftest.py	Populate bgp_vrf, bgp_prime_asn, intf_map on each EosHost from multi_vrf_data in nbrhosts; only converge the topology when the neighbor type is cEOS so SONiC-VS / cisco / csonic runs keep historical behavior.	if multi_vrf_peer: / neighbor_type in (eos, ceos)
ansible/roles/eos/templates/ceos_converged.j2 (new)	Single shared converged startup-config, rendered as a pure function of the converger output (convergence_data) plus per-peer configuration — independent of base topo / role.	topo_is_multi_vrf
ansible/roles/eos/templates/{t0-64-32-leaf, t2-leaf, t2-core, dpu-tor, t1-8-lag-spine, t1-8-lag-tor}.j2	Include ceos_converged.j2 when converged; the existing stock body is preserved unchanged in the {% else %} branch.	when: topo_is_multi_vrf
ansible/ceos_topo_converger.py	Select one prime per role (not per role+ASN). Each merged sub-peer is a VRF with its own local-as, so a single prime correctly serves sub-peers with different ASNs and the prime count stays minimal (e.g. dpu → 1, t1-lag/t1-lag-vpp/t2 → 2). Also preserve disabled_host_interfaces; surface the converged front-panel count as max_fp_num_provided; and preserve the topology-level dut_num / topo_type so multi-linecard minigraph generation sizes the per-DUT interface lists correctly (fixes the deploy-mg IndexError).	n/a — converger only runs in converged mode
ansible/roles/vm_set/tasks/main.yml	Apply the converger's max_fp_num_provided override for every vm_set action (not just start), so the add_topo bridge/veth creation matches the converged front-panel count (fixes the add-topo "Too many vlans").	when: max_fp_num_provided is defined
ansible/roles/eos/tasks/ceos_config.yml + ansible/roles/eos/templates/ceos_bp_compat.j2 (new)	Append a stock-compat backplane shim (untagged VLAN 1 on the trunk, Vlan1 SVI in the prime VRF carrying the host bp_interface IP, advertise that subnet via BGP).	when: topo_is_multi_vrf and configuration[hostname]['bp_interface'] is defined
ansible/roles/vm_set/library/vm_topology.py	Assign the legacy backplane IP (ptf_bp_ip[v6]_addr) to the PTF parent backplane in addition to the per-VRF sub-interfaces.	Only the if is_multi_vrf: branch of add_bp_port_with_vlans_to_docker()
tests/bgp/test_bgp_bbr.py	One-line change: derive the exabgp port from get_neighbor_exabgp_vm_offset() instead of the raw (prime) offset.	helper is a no-op on stock
tests/vxlan/test_vxlan_route_advertisement.py	One-line fix for the pre-existing crash: the scale retry used self.vxlan_test_setup['t2']['host'] (indexing a list with a string) → corrected to the t2device['host'] loop variable.	not converged-specific (genuine bugfix)
ansible/testbed-cli.sh	Only converge the topology when deploying with -k ceos; non-cEOS vm types skip convergence.	if vm_type == ceos
ansible/vtestbed.yaml	Enable use_converged_peers: True on the virtual testbeds; convergence then applies only to their cEOS deployments via the gates above.	n/a

Verification

Validated on virtual testbeds (cEOS neighbors), confirming the converged render and the gating:

• dpu — converges to a single prime hosting both ToR VRFs with per-VRF local-as (ASN 65200 and 65201). add-topo + deploy-mg succeed; the DUT establishes both BGP sessions and exchanges routes; bgp/test_bgp_fact.py passes including the pre-test BGP sanity check that previously failed.
• multi-asic (t1-8-lag) — the shared template now renders the per-VRF config, so the ns-<vrf> Linux namespaces are created on the prime (the missing namespace that previously made snmp/test_snmp_loopback.py fail with "Cannot open network namespace").
• t1-lag / t1-lag-vpp / t2 — full add-topo → deploy-mg → test cycle now succeeds end-to-end on virtual testbeds; bgp/test_bgp_fact.py passes with every DUT BGP session established, including the LAG (Port-Channel) neighbors. One-prime-per-role keeps the prime count low (add-topo no longer exhausts memory); the max_fp_num_provided override now applied at add_topo creates the correct number of front-panel bridges (no more "Too many vlans"); and preserving dut_num lets multi-linecard t2 minigraph generation succeed (no more deploy-mg IndexError). On t1-lag-vpp the LAGs come up over the VPP dataplane (SONiC teamd LACP punted via linux-cp) with no platform-image change required.
• Per-test triage (cEOS neighbors):
  • bgp/test_bgp_bbr.py (t1-lag) — passes; the exabgp-offset helper resolves the port to the merged neighbor's original instance.
  • vxlan/test_vnet_bgp_route_precedence.py (test_vnet_route_after_bgp + multi_flap, t1-lag-vpp) — 2 passed; the VRF-scoped router bgp/network and de-collided loopback let the DUT learn the advertised route.
  • vxlan/test_vxlan_route_advertisement.py::test_scale_route_advertisement_with_community (v4_in_v4, v6_in_v4, t1-lag-vpp) — 2 passed. With VRF-scoping, the scale read show ip bgp community <c> vrf <vrf> | grep returns the full 4000-route set, where the un-scoped read returned 0.
• Non-cEOS deployments (SONiC-VS / cisco / csonic) skip convergence entirely (gated in both testbed-cli.sh and conftest.py) and remain byte-identical to historical behavior.
• Stock topology paths are unchanged when topo_is_multi_vrf / is_multi_vrf is false, and when self.bgp_vrf is None — verified by code inspection of the gating conditions, by unit-exercising the new eos.py rewrites (stock inputs pass through unchanged), and by Jinja-parsing every wrapped template.

Any platform specific information?

Converged peer model only; cEOS neighbors only. Non-cEOS neighbor types (SONiC-VS / cisco / csonic) are explicitly excluded from convergence at both the deploy (testbed-cli.sh) and test (conftest.py) layers, so they keep their existing behavior.

[yijingyan2]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

[auspham]

/azpw run

[mssonicbld]

⚠️ Notice: /azpw run only runs failed jobs now. If you want to trigger a whole pipline run, please rebase your branch or close and reopen the PR.
💡 Tip: You can also use /azpw retry to retry failed jobs directly.

Retrying failed(or canceled) jobs...

[mssonicbld]

No Azure DevOps builds found for #24963.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[auspham]

1st stage: all passed only t0 multi_vrf enabled

Will toggle on for other topology

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[auspham]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[auspham]

/azpw run

[mssonicbld]

⚠️ Notice: /azpw run only runs failed jobs now. If you want to trigger a whole pipline run, please rebase your branch or close and reopen the PR.
💡 Tip: You can also use /azpw retry to retry failed jobs directly.

Retrying failed(or canceled) jobs...

[mssonicbld]

No failed(or canceled) stages or jobs found in the most recent build 1135951.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[auspham]

all tests passing

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[auspham]

/azpw retry

[mssonicbld]

Retrying failed(or canceled) jobs...

[mssonicbld]

Retrying failed(or canceled) stages in build 1136471:

✅Stage Pre_test:

• Job Markers Check: retried.
• Job Detect skip_expiry changes: retried.

[lunyue-ms]

The Verification section covers dpu, t1-lag, t1-lag-vpp, t2, and t0 passed nightly, but t0-64-32, and dual-t0 don't seem to be exercised yet. Can we confirm their tests pass under convergence?

[auspham]

Hi @lunyue-ms, t0-64-32 is actually t0-sonic. For sonic we dont have support for VRF. For dualtor it actually unstable even with ceos so I'm still trying to fix that first. Please let me know if you have any comments on the code meanwhile. thanks

[lunyue-ms]

Thanks for the confirm. It looks good to me. I used agent to understand your modification. It's promising to decrease the cEOS instance number now.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[yejianquan]

LGTM