Skip to content

Commit 6bf2f96

Browse files
SpiritreaderSpiritreader
committed
feat!: carry both legs on kuport's own link, and restructure the docs
Single joins Multi on the decoupled inbound path. An accepting node holding no pod translates a request to the far end of its own return link rather than to the pod address, so the request rides kuport's encapsulation in both modes. The gate moved from servingMode to whether a link exists at all. Two cluster requirements go with it. The CNI no longer has to tunnel, because neither leg uses its encapsulation, and pod addresses no longer have to be routable between nodes, because nothing outside a node ever addresses its pods. The TunnelModeRequired condition is removed; the VXLAN port collision check stays, since two devices still cannot share a VNI and port. The conntrack cost quoted against doing this turned out to be nothing. Both nodes already held an entry per flow: measured on the cluster, the pod's node carried an [UNREPLIED] record created by the reply, because the request had reached the pod through the CNI without touching netfilter. That entry is now a established flow carrying the translation instead. Docs: spec.md and spec-decoupled-inbound.md are folded into overview, api, architecture, datapath, decisions and development, indexed by docs/README.md. operations.md is renamed troubleshooting.md. spec-decoupled-inbound.md was a proposal and is implemented, so its design lives in datapath.md and its reasoning in decisions.md alongside the four approaches measurement closed.
1 parent 04784fe commit 6bf2f96

16 files changed

Lines changed: 1319 additions & 1346 deletions

‎README.md‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -45,12 +45,18 @@ If you are considering it: install on a staging cluster, walk
4545
[docs/integration.md](docs/integration.md) end to end, and expect to file the
4646
first issues.
4747
48-
- [docs/spec.md](docs/spec.md): the design, the datapath with its
49-
measurements, diagrams, and the closed decisions.
50-
- [docs/operations.md](docs/operations.md): a mapping is not working, and
51-
every condition reason means something.
52-
- [docs/integration.md](docs/integration.md): installing kuport into a
53-
specific cluster, written for whoever does the installing.
48+
[docs/README.md](docs/README.md) indexes all of it. The ones most people want:
49+
50+
- [docs/overview.md](docs/overview.md): the problem, the goals, and what a
51+
cluster has to provide.
52+
- [docs/api.md](docs/api.md): both kinds field by field, and what status
53+
reports.
54+
- [docs/datapath.md](docs/datapath.md): the packet's path, every rule on it
55+
with the measurement that confirmed it, and the serving modes.
56+
- [docs/troubleshooting.md](docs/troubleshooting.md): a mapping is not working,
57+
and what every condition reason means.
58+
- [docs/integration.md](docs/integration.md): installing kuport into a specific
59+
cluster, written for whoever does the installing.
5460
5561
## Install
5662

‎docs/README.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# kuport documentation
2+
3+
kuport delivers a TCP or UDP port from one node's addresses to a pod, keeping
4+
the client's source address, and lets the workload ask for that itself.
5+
6+
Start with [overview.md](overview.md) if you have not met it before.
7+
8+
## By what you are doing
9+
10+
| You want to | Read |
11+
| --- | --- |
12+
| understand what this is and whether it fits your cluster | [overview.md](overview.md) |
13+
| write a PortMapClass or a PortMap | [api.md](api.md) |
14+
| know why a packet goes where it goes | [datapath.md](datapath.md) |
15+
| find out why a mapping is not working | [troubleshooting.md](troubleshooting.md) |
16+
| install it into a cluster | [integration.md](integration.md) |
17+
| know why the design is shaped this way | [decisions.md](decisions.md) |
18+
| understand how the agents divide the work | [architecture.md](architecture.md) |
19+
| build, test or release the agent | [development.md](development.md) |
20+
21+
## By document
22+
23+
**[overview.md](overview.md)** states the problem, the goals and non-goals, and
24+
what a cluster has to provide.
25+
26+
**[api.md](api.md)** covers both kinds field by field, how a mapping's
27+
interfaces resolve against each node's, and what the two status blocks report.
28+
29+
**[architecture.md](architecture.md)** is the DaemonSet with no central
30+
controller: how every agent computes the same answer, which one writes a given
31+
status, and how link slots are claimed through the API server.
32+
33+
**[datapath.md](datapath.md)** is the packet's path and every rule on it, the
34+
same-node and cross-node shapes, the serving modes with an example class for
35+
each, the return link, and MTU.
36+
37+
**[decisions.md](decisions.md)** records the closed questions with the
38+
measurement that settled each one, including the four approaches to cross-node
39+
`Multi` that were closed before the one that works.
40+
41+
**[troubleshooting.md](troubleshooting.md)** is for a mapping that is not
42+
working: reading status, the counters, the failures that leave no log line, the
43+
host firewall, MTU, and what happens to host state when the agent moves.
44+
45+
**[integration.md](integration.md)** brings kuport into a cluster.
46+
47+
**[development.md](development.md)** is the repo layout, the implementation
48+
notes, the test tiers and the release pipeline.
49+
50+
## History
51+
52+
Two documents were folded into these and removed. `spec.md` became
53+
[overview.md](overview.md), [api.md](api.md),
54+
[architecture.md](architecture.md), [datapath.md](datapath.md),
55+
[decisions.md](decisions.md) and [development.md](development.md).
56+
`spec-decoupled-inbound.md` proposed carrying the request on kuport's own link;
57+
it was implemented in v0.3.6 and v0.4.0, so its design lives in
58+
[datapath.md](datapath.md) and its reasoning in
59+
[decisions.md](decisions.md#carrying-both-legs-on-kuports-own-link).

‎docs/api.md‎

Lines changed: 226 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,226 @@
1+
# API
2+
3+
Group `kuport.wlz.li`, version `v1alpha1`. Two kinds: a class the admin writes
4+
and a mapping the workload writes.
5+
6+
## PortMapClass
7+
8+
Cluster-scoped. The admin writes it. It says which nodes accept traffic, on
9+
which interfaces, which ports may be asked for, and which namespaces may ask.
10+
11+
```yaml
12+
apiVersion: kuport.wlz.li/v1alpha1
13+
kind: PortMapClass
14+
metadata:
15+
name: public
16+
spec:
17+
nodes:
18+
worker-1:
19+
interfaces:
20+
- wt0
21+
- enp1s0
22+
worker-2:
23+
interfaces:
24+
- wt0
25+
- eth0
26+
ports:
27+
min: 1024
28+
max: 65535
29+
reserved: [80, 443]
30+
namespaceSelector:
31+
matchLabels:
32+
kuport.wlz.li/public: allowed
33+
returnPath:
34+
mode: Vxlan
35+
vxlan:
36+
vni: 4242
37+
port: 4790
38+
subnet: 169.254.77.0/24
39+
```
40+
41+
| Field | Type | Meaning |
42+
| --- | --- | --- |
43+
| nodes | map of node name to object | Nodes that may accept traffic for this class: its accepting nodes. A name with no Node object in the cluster is skipped. Required, at least one entry. |
44+
| nodes.<name>.interfaces | list of string | Interface names that node's DNAT rules match on. One rule per interface. Naming them per node lets one class cover nodes whose NICs are named differently. Required, at least one. |
45+
| servingMode | enum | `Single` or `Multi`: how many accepting nodes program a mapping. Default `Single`. See [serving modes](datapath.md#serving-modes). |
46+
| ports.min, ports.max | int | The range a PortMap may ask for. Defaults 1, 65535. |
47+
| ports.reserved | list of int | Ports the admin keeps back. A PortMap naming one is rejected in status. |
48+
| namespaceSelector | label selector | Which namespaces may reference this class. Empty selects every namespace. |
49+
| returnPath.mode | enum | `Vxlan` or `None`. `None` refuses any mapping that would need a return link. |
50+
| returnPath.vxlan.vni | int | Base VXLAN network identifier for the links this class builds. Each slot takes the base plus its own number, because the kernel keys a device by VNI and port. |
51+
| returnPath.vxlan.port | int | UDP port for the links. Must differ from the CNI's, which is 8472 for Cilium. Default 4790. |
52+
| returnPath.vxlan.subnet | CIDR | Link addresses are allocated from here, a /31 per node pair (RFC 3021 point-to-point), so the default gives 128 slots. The slot for a pair is a recorded claim in status.links rather than a computed hash. Default 169.254.77.0/24. |
53+
54+
Two classes whose accepting nodes overlap need their own `vni` and `subnet`. A
55+
class numbers its slots from zero, so two classes sharing a node would both ask
56+
for the base VNI and the first /31 there, and the kernel refuses the second of
57+
each.
58+
59+
## PortMap
60+
61+
Namespaced. The workload writes it, beside its Service.
62+
63+
```yaml
64+
apiVersion: kuport.wlz.li/v1alpha1
65+
kind: PortMap
66+
metadata:
67+
name: gameserver
68+
namespace: games
69+
spec:
70+
className: public
71+
protocol: UDP
72+
port: 3000
73+
serviceRef:
74+
name: gameserver
75+
port: game
76+
```
77+
78+
| Field | Type | Meaning |
79+
| --- | --- | --- |
80+
| className | string | The PortMapClass this asks for. Required, immutable. |
81+
| protocol | enum | TCP or UDP. Required, immutable. |
82+
| port | int | The port clients dial on an accepting node, and the port the pod receives on. With endPort set, the first port of the range. Required, immutable. |
83+
| endPort | int | Last port of an inclusive range starting at port. Omit for a single port. Immutable together with port; the API rejects endPort below port. One nftables `dport <first>-<last>` match covers a whole range. |
84+
| interfaces | list of string | The interfaces this mapping binds on, narrowing what the class gives each node. Empty selects every interface the class gives the node. Mutable. |
85+
| serviceRef.name | string | A Service in the same namespace. Required. |
86+
| serviceRef.port | string | The named port on that Service. Required. |
87+
88+
The Service may be any type, ClusterIP included. It exists so the agent has
89+
endpoints to follow and a named port to resolve. kuport never touches it.
90+
91+
The immutable fields are immutable because changing them is indistinguishable
92+
from deleting one mapping and creating another, and the reconcile is simpler if
93+
it never has to unwind a half-changed mapping.
94+
95+
## Choosing interfaces
96+
97+
A mapping's interfaces resolve against each accepting node's own list, so one
98+
list covers nodes whose NICs are named differently. Given a class that gives
99+
worker-1 wt0 and enp1s0, and worker-2 wt0 and eth0:
100+
101+
| The mapping asks for | worker-1 binds | worker-2 binds |
102+
| --- | --- | --- |
103+
| nothing | wt0, enp1s0 | wt0, eth0 |
104+
| wt0, enp1s0, eth0 | wt0, enp1s0 | wt0, eth0 |
105+
| wt0 | wt0 | wt0 |
106+
| enp1s0 | enp1s0 | nothing |
107+
108+
Asking for wt0 alone keeps a mapping on the overlay. Asking for the LAN
109+
interfaces as well as wt0 reaches clients on both.
110+
111+
An interface name that goes nowhere is reported, in one of three places
112+
depending on where the mistake is:
113+
114+
| Mistake | Reported as | Where |
115+
| --- | --- | --- |
116+
| The mapping names an interface no node in the class carries | Accepted=False, InterfaceNotInClass | the PortMap |
117+
| The mapping names interfaces the class carries, none of them on the node that ended up serving | Programmed=False, NoInterfaceOnNode | the PortMap |
118+
| The class gives a node an interface the host does not have | the node row is not ready, `interface eth0 not present` | the PortMapClass, and as Programmed=False, NodeNotReady on each mapping it serves |
119+
120+
A name some node carries is accepted, since binding on a subset of the nodes is
121+
the point of the field. The second row catches the case that gets through: a
122+
mapping admitted by the class check whose serving node has none of the names it
123+
asked for, which would otherwise program nothing in silence.
124+
125+
## PortMap status
126+
127+
```yaml
128+
status:
129+
endpoint:
130+
node: worker-b
131+
address: 10.244.18.107
132+
published:
133+
- node: edge-a
134+
interface: enp1s0
135+
address: 203.0.113.9
136+
- node: edge-a
137+
interface: wt0
138+
address: 100.64.93.143
139+
observedGeneration: 3
140+
conditions:
141+
- type: Accepted
142+
status: "True"
143+
reason: Valid
144+
- type: Programmed
145+
status: "True"
146+
reason: AllNodesReady
147+
```
148+
149+
`published` is what a person needs: the addresses this port answers on right
150+
now, one row per class interface on the mapping's serving node. It is what
151+
`kubectl get portmap` prints in its wide columns.
152+
153+
The serving node writes the mapping's status, and every published row names one
154+
of its own interfaces, so it resolves each address off its own host. A row whose
155+
interface has not resolved yet carries an empty address and fills in on the next
156+
pass.
157+
158+
| Condition | True when | Notable false reasons |
159+
| --- | --- | --- |
160+
| Accepted | the class exists, selects this namespace, the port range is inside the class range and unreserved, and no earlier mapping holds it | `ClassNotFound`, `NamespaceNotSelected`, `PortOutOfRange`, `PortReserved`, `PortConflict`, `InvalidPortRange` |
161+
| Programmed | the serving node has written its rules and its class status row reports ready, and for a pod on another node the return link has landed with usable addresses at both ends | `NoReadyEndpoint`, `ReturnPathUnavailable`, `NodeNotReady` |
162+
163+
`InvalidPortRange` is the ceiling on the CEL rule that endPort must not be below
164+
port: an update the API server let through still reports here.
165+
166+
`Programmed` gates on the participants: the serving node's row in the class
167+
status, and for a remote pod the landed link slot. Under Single the other
168+
accepting nodes hold nothing for the mapping, so their rows gate nothing. Until
169+
a gate clears, the status names what is missing, and the level-driven reconcile
170+
heals it.
171+
172+
## Class status
173+
174+
The class reports each selected node's readiness verdict and which return links
175+
hold an address slot.
176+
177+
```yaml
178+
status:
179+
links:
180+
- key: edge-a/worker-b
181+
peers: [edge-a, worker-b]
182+
subnet: 169.254.77.2/31
183+
slot: 1
184+
nodes:
185+
- name: edge-a
186+
ready: true
187+
underlayMTU: 1400
188+
linkMTU: 1350
189+
addresses:
190+
enp1s0: 203.0.113.9
191+
wt0: 100.64.93.143
192+
- name: worker-c
193+
ready: false
194+
message: interface wt0 not present
195+
observedGeneration: 2
196+
conditions:
197+
- type: Ready
198+
status: "True"
199+
reason: Valid
200+
```
201+
202+
One `links` entry per node pair that has, or recently had, a return link. The
203+
`unusedSince` field appears when no PortMap needs the pair and is set to the
204+
time that became true; an entry unused for 24h is dropped. The slot fixes the
205+
routing table (`200 + slot`), the packet mark (`0x6b700000 | slot`) and the
206+
link's VNI, so a claim is never renumbered while a link uses it.
207+
208+
One `nodes` row per accepting node, written by that node's agent, carrying the
209+
MTU numbers and interface addresses it read from the host. The row is ready when
210+
every interface the class gives that node resolves to an address there;
211+
otherwise it carries a message naming the first interface that does not. The
212+
class names each node's interfaces, so an interface that fails to resolve is a
213+
mistake in the class rather than a node that lacks the NIC, and the row says so
214+
rather than skipping it.
215+
216+
`linkMTU` is that node's own figure, `underlayMTU` less the encapsulation. A
217+
link device takes the smaller of its two ends' figures, so a link to a peer on a
218+
thinner underlay carries less than either row suggests. [MTU in
219+
troubleshooting.md](troubleshooting.md#mtu) has the reading.
220+
221+
| Condition | True when | Notable false reasons |
222+
| --- | --- | --- |
223+
| Ready | the subnet parses, has a free slot, and the link port does not collide with the CNI's | `VxlanPortConflict`, `SubnetExhausted`, `InvalidSubnet` |
224+
225+
`TunnelModeRequired` was a fourth reason until v0.4.0, when both legs moved onto
226+
kuport's own return link and the CNI's routing mode stopped deciding anything.

0 commit comments

Comments
 (0)