Skip to content

Commit cf34993

Browse files
committed
docs: the Multi non-goal says which side it spreads
A reader hitting the load-balancing non-goal had no way to tell entry-point failover from pod load balancing, and read Multi behind a routed address as the latter. The line now names the distinction: Multi spreads the accepting side, every accepting node forwards to the same one endpoint.
1 parent 896db30 commit cf34993

1 file changed

Lines changed: 79 additions & 2 deletions

File tree

docs/spec.md

Lines changed: 79 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,10 @@ may send.
5151

5252
- HTTP routing or TLS. An ingress controller already does that better.
5353
- Load balancing one port across pods on several nodes. One endpoint is chosen
54-
and reprogrammed when it goes away.
54+
and reprogrammed when it goes away. `servingMode: Multi` spreads the accepting
55+
side rather than the serving side: every accepting node programs the mapping,
56+
so a routed address survives one of them going away, and all of them forward
57+
to that same one endpoint.
5558
- IPv6 in the first version. Every rule here is `table ip`.
5659
- Admission webhooks. Conflicts are reported in status; the reasoning is under
5760
Decisions.
@@ -120,7 +123,7 @@ spec:
120123
| ports.min, ports.max | int | The range a PortMap may ask for. Defaults 1, 65535. |
121124
| ports.reserved | list of int | Ports the admin keeps back. A PortMap naming one is rejected in status. |
122125
| namespaceSelector | label selector | Which namespaces may reference this class. Empty selects every namespace. |
123-
| returnPath.mode | enum | `Vxlan` or `None`. `None` refuses any mapping whose pod is on another node. |
126+
| returnPath.mode | enum | `Vxlan` or `None`. `None` refuses any mapping that would need a return link. See returnPath mode below. |
124127
| returnPath.vxlan.vni | int | VXLAN network identifier for the links this class builds. |
125128
| returnPath.vxlan.port | int | UDP port for the links. Must differ from the CNI's, which is 8472 for Cilium. Default 4790. |
126129
| returnPath.vxlan.subnet | CIDR | Link addresses are allocated from here, a /31 per node pair (RFC 3021 point-to-point), so the default gives 128 slots. The slot for a pair is a recorded claim in status.links, not a computed hash. Default 169.254.77.0/24. |
@@ -766,6 +769,80 @@ is conntrack-based, so a reply after the entry ages out is not translated back
766769
and the client discards it. `Multi` extends that dependency to the pod's node
767770
rather than introducing a new kind of failure.
768771

772+
### returnPath mode
773+
774+
`Vxlan` builds the return link described under The return link. `None` builds no
775+
link and refuses any mapping that would need one.
776+
777+
A mapping needs a return link when the pod sits on a node other than the one
778+
holding the DNAT rule. The DNAT happens on the programming node, and the
779+
conntrack entry that records the address the client originally dialed lives on
780+
that node alone. A reply leaving the pod's node by its own uplink carries the pod
781+
address as source, and the client discards it as spoofed. The reply has to arrive
782+
back at the node that translated the request, which is what the link is for.
783+
784+
```mermaid
785+
sequenceDiagram
786+
autonumber
787+
participant C as Client
788+
participant E as Programming node edge-a
789+
participant T as Pod node worker-b
790+
participant P as Pod at 10.244.18.107
791+
C->>E: request: dst 203.0.113.9:3000, src 198.51.100.7
792+
Note over E: kup-pre DNAT rewrites the dst to the pod address. The conntrack entry that remembers 203.0.113.9:3000 exists on edge-a and nowhere else.
793+
E->>T: Cilium vxlan carries it, inner src still the client
794+
T->>P: the pod reads src 198.51.100.7
795+
P-->>T: reply: src 10.244.18.107:3000, dst 198.51.100.7
796+
Note over T: with no return link, the default route sends the reply out worker-b's own uplink
797+
T-->>C: src 10.244.18.107, an address the client never dialed
798+
Note over C: discarded as spoofed, and the connection never completes
799+
```
800+
801+
`None` refuses the mapping before any of that is programmed, so the port stays
802+
closed and the PortMap status says why.
803+
804+
The refusal is decided from the same shared inputs on every agent. Each agent
805+
computes the mapping's programming nodes and compares them against the node
806+
holding the chosen endpoint. Under `Single` the programming set is the serving
807+
node alone, and the serving node is the endpoint's node whenever that node
808+
accepts the class. Under `Multi` the programming set is every accepting node.
809+
810+
```mermaid
811+
flowchart TD
812+
A["the chosen endpoint: one pod, picked identically by every agent"] --> B{"servingMode"}
813+
B -- Single --> C["programming nodes: the serving node, which is the pod's node whenever that node accepts the class"]
814+
B -- Multi --> D["programming nodes: every accepting node"]
815+
C --> E{"does every programming node hold the chosen pod?"}
816+
D --> E
817+
E -- yes --> F["no return link is needed: the reply passes back through the conntrack entry that translated the request"]
818+
E -- no --> G{"returnPath.mode"}
819+
G -- Vxlan --> H["a link, a /31 slot and a routing table for each programming node that lacks the pod"]
820+
G -- None --> I["Programmed=False with ReturnPathUnavailable, the programmer set is cleared, and no node writes a DNAT rule"]
821+
```
822+
823+
| servingMode | Where the chosen pod sits | `Vxlan` | `None` |
824+
| --- | --- | --- | --- |
825+
| `Single` | on the serving node | programmed, no link built | programmed, no link built |
826+
| `Single` | on a node outside the class's accepting nodes | programmed, one link | refused |
827+
| `Multi`, one accepting node | on that node | programmed, no link built | programmed, no link built |
828+
| `Multi`, several accepting nodes | anywhere | programmed, one link per accepting node that lacks the pod | refused |
829+
830+
Under `Multi` the comparison runs over every accepting node against the one
831+
chosen endpoint, so a class with several accepting nodes refuses every mapping
832+
while `None` is set. A DaemonSet does not change that outcome. There is no
833+
per-node endpoint choice to fall back on: chooseEndpoint returns a single pod for
834+
the whole mapping, which is what stops two agents from programming different
835+
pods, and every programming node DNATs to that one address.
836+
837+
`None` is the setting for a class whose pods already sit on its accepting nodes,
838+
which under `Single` means a DaemonSet, or a workload pinned to an accepting node
839+
by its own scheduling constraints. Only one of a DaemonSet's pods receives
840+
traffic for a given mapping; the rest are there so the endpoint choice always
841+
finds a candidate on an accepting node. A rescheduling that moves the pod off the
842+
accepting set then surfaces as a refused mapping carrying ReturnPathUnavailable.
843+
Under `Vxlan` the same rescheduling builds a link and the mapping keeps working,
844+
which is the answer a class wants when pod placement is free.
845+
769846
## Repo layout
770847

771848
```

0 commit comments

Comments
 (0)