crazy multiport stuff

[JackDoan]

I had clod make a multi-tunnel version of multiport just to play around. It benchmarks very well for me!

TLDR: each routine now also gets its own full underlay flow on a different port. That way, flows get steered by 5-tuple automatically. There's coordination in the handshake manager for first handshaking on the "base" port advertised by the lighthouse, and then adding more "lanes" between the hosts with additional handshakes.

Here's what clod thinks you should know:

Reviewer Guide: Multiport Lanes

The problem

A Nebula tunnel is one UDP "flow": one source IP/port talking to one
destination IP/port. Every router, NIC, and load balancer on the path uses
that 4-tuple to pick a path and a receive queue. So no matter how many TCP
streams you push through the tunnel, they all squeeze through one path and
one receive queue on the far side. On our bench, 4 parallel iperf streams
scored exactly the same as 1 stream.

The idea

Give each pair of hosts several UDP flows instead of one. When multiport is
on, a host with routines: 4 binds 4 consecutive UDP ports (listen.port
through listen.port+3) instead of sharing one port with SO_REUSEPORT.
After the normal tunnel ("base tunnel") comes up, each side opens one extra
tunnel per routine, called a lane. Lane i handshakes from local socket i
to the peer's port base+i. Traffic read from tun queue i uses lane i; the
kernel already pins each flow to one tun queue, so a flow stays on one lane
and packets never reorder.

Each lane is a complete, normal Noise tunnel — its own keys, its
own nonce counter, its own replay window. That is the key design choice.
The alternative (upstream PR #768) spreads one tunnel's packets across many
source ports with a raw socket. But then all those paths share one replay
window: if one path is slower than another, the slow path's packets fall
outside the window and get dropped as replays. Separate tunnels also mean:

• RX needs no changes. Packets are matched by the index in the header,
  never by source address, so lanes "just work" on receive.
• Each lane handshakes over its own 4-tuple, so firewalls and NAT see a
  normal connection per lane. No roaming hacks needed.
• Each lane's replay window is only touched by one goroutine (lane i's
  packets arrive only on socket i), so no lock contention.

Costs: a few extra handshakes and keepalives per peer, and more hostmap
bookkeeping (the risky part — see "What to review closely").

Compatibility

Negotiation uses handshake payload fields 6 and 7, which were reserved
upstream. Vanilla peers skip unknown fields (there's a test proving the
encoding is byte-identical when multiport is off), so a vanilla peer just
gets one normal tunnel. All control traffic — handshakes, lighthouse,
punching, relays — stays on the base tunnel and base port. If lanes can't
form (blocked ports, NAT), traffic falls back to the base tunnel.

multiport.enabled defaults on and quietly turns itself off when the
setup can't support it (1 routine, non-Linux, no multi-reader UDP). It
can't hard-error because dnclient owns the config. A dynamic listen.port: 0 works too: socket 0 binds a random port, then we claim the next N-1
ports, retrying with a fresh random port if the range is taken.

Map of the changes

Area	What changed
handshake/payload.go, machine.go	New LaneDetails{PortCount, BasePort, LaneIndex} in fields 6/7. NewMachine takes it; Result returns the peer's.
main.go	Config gating, capability probe, per-port bind loop with dynamic-port retry.
hostmap.go	laneState on the base HostInfo: txLanes[i] (atomic, read lock-free on TX), pending/backoff bookkeeping, peer-lane list. Lanes live in Indexes/RemoteIndexes but never Hosts. Deleting a base cascades to its lanes; deleting a lane clears only its slot.
handshake_manager.go	StartLaneHandshake / handleOutboundLane (single pinned target, sent from the lane's socket, no lighthouse); completeLaneResponder (attach by certified vpn address, per-lane replay dedup); EnsureLanes (fill empty slots with 5s→60s backoff). Handshake replies now egress the socket the request arrived on.
connection_manager.go	Lanes get keepalives (their death detector) but never become primary, never swap/migrate/rehandshake/punch. A base with live lanes can't be reaped as "inactive". ensureLanes runs on the existing per-tunnel tick.
inside.go, interface.go	HostInfo.sockIdx decides which socket every packet leaves from (0 = base = old behavior). Per-routine txQueue{lane, base}: lane data on socket i, base+relay data on socket 0, flushed base-first. The hot-path lane check is one atomic load.
outside.go	ViaSender.SockIdx carries the arrival socket; recv_error replies use it (the peer's anti-spoof check compares source addresses).

Also fixed a latent upstream bug: the pending-handshake table deleted
entries by address without checking they belonged to the hostinfo being
deleted, which could evict an unrelated in-flight handshake.

What to review closely

1. The lock-free publish contract. txLanes[i] is only Stored after
   the lane's ConnectionState is fully built, and cleared with a CAS. The
   TX path (sendInsideMessage) loads it with no lock. Check the ordering.
2. Lock order. Always hostmap lock → laneState lock, or laneState
   alone. The base-delete cascade snapshots lanes under the laneState lock,
   then deletes; verify no path inverts this.
3. sockIdx everywhere. Any packet on a lane that leaves from the
   wrong socket makes the peer see a roam or reject a reply. sendNoMetrics
   derives it from the hostinfo; grep for any send path that doesn't.
4. Delete paths are identity-checked so double-deletes and races are
   no-ops. unlockedDeleteLane, noteOwnedLaneDeath, the vpnIps fix.
5. Base rehandshake. A new base gets a fresh laneState; the old
   generation's lanes die with the old base. Parent pointers keep the
   cascades from crossing generations.

Evidence

• Unit tests: lanes_test.go, handshake/machine_lanes_test.go, payload
  round-trip + vanilla byte-compat tests. Full suite green.
• Live loopback pair: lanes form instantly, 50s with zero roams, zero
  deaths, zero churn; vanilla interop clean.
• Bench (jackdesk↔bigborg, A/B/A): single flow unchanged (~7.5 FWD / ~11
  REV Gbps both ways). 4 streams: 7.5 → ~20 Gbps FWD (+160%), 11 → 22
  Gbps REV (+100%). With multiport off, 4 streams score the same as 1 —
  the exact bottleneck this removes. tcpdump confirmed traffic on 3 lane
  ports + base. Works over the IPv6 underlay.

Known gaps

• No multiport e2e tests yet — the e2e harness assumes one socket per node.
• Lanes need the peer's port range reachable; behind unfriendly NAT they
  simply never form and everything rides the base tunnel.
• multiport.lanes (running fewer lanes than routines) is implemented but
  lightly tested.