Skip to content

Commit 7e9444d

Browse files
OtowoSamuelOtowoSamuel
authored
Merge pull request #1029 from godamongstmen897/feature/stellar-k8s-ops-compliance
docs/infra: deploy cluster troubleshooting matrices, enterprise compl…
2 parents 6c86852 + f0f11c1 commit 7e9444d

10 files changed

Lines changed: 819 additions & 20 deletions

File tree

‎README.md‎

Lines changed: 0 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -609,22 +609,6 @@ make generate-api-docs
609609

610610
For detailed instructions on setting up a local development environment, building the project, running tests, and managing Kubernetes resources locally, please refer to the **[Development Guide](DEVELOPMENT.md)**.
611611

612-
### Quick Start Reference
613-
614-
```bash
615-
# Setup development environment (includes pre-commit hooks)
616-
make dev-setup
617-
618-
# Run quick checks (format & cargo check)
619-
make quick
620-
621-
# Run all tests
622-
make test
623-
624-
# Build release binaries
625-
make build
626-
```
627-
628612
### Reconciler fuzzing
629613

630614
To ensure the operator never panics under malformed or extreme inputs, the reconciler is fuzzed with random `StellarNodeSpec` mutations and event sequences (proptest). Run the fuzzer locally:

‎docs/networking/index.md‎

Lines changed: 251 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,251 @@
1+
# Network Configuration & CNI/BGP Integration Guide
2+
3+
This guide describes the network architecture, container network interface (CNI) configuration, border gateway protocol (BGP) routing, and service mesh integration for deploying **Stellar-K8s** in high-performance environments.
4+
5+
---
6+
7+
## 1. Network Architecture & Topology
8+
9+
Stellar Core nodes communicate using a custom peer-to-peer (P2P) protocol over TCP port `11625`. The Rest API (Horizon) and Soroban RPC nodes communicate over HTTP/HTTPS (ports `8000` and `8080`).
10+
11+
### 1.1 Cluster Traffic Topology
12+
```text
13+
┌──────────────────────────────────────────┐
14+
│ Internet │
15+
└──────────────────┬───────────────────────┘
16+
│
17+
│ (Port 11625 TCP P2P)
18+
▼
19+
┌────────────────────────────┐
20+
│ MetalLB / Load │
21+
│ Balancer │
22+
└─────────────┬──────────────┘
23+
│
24+
┌────────────────────────┼────────────────────────┐
25+
│ (Port 11625 TCP) │ (Port 8000/8080 HTTP) │ (Prometheus Scraping)
26+
▼ ▼ ▼
27+
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
28+
│ Stellar Core │ │ Horizon API │ │ Prometheus │
29+
│ (Validator Pod) │ │ (RPC Pod) │ │ Server │
30+
└──────────────────┘ └──────────────────┘ └──────────────────┘
31+
```
32+
33+
---
34+
35+
## 2. CNI Plugin Integration
36+
37+
Stellar-K8s supports advanced networking through industry-standard CNIs.
38+
39+
### 2.1 Calico CNI
40+
Calico provides high-performance networking and rich network security policies using standard Linux iptables or IPVS.
41+
- **Configuring Multi-Cluster Networking**: Enable Calico's IPpool encapsulation (VXLAN or IP-in-IP) for cross-subnet overlay routing.
42+
- **GlobalNetworkPolicy**: Define global policies to allow Stellar P2P ports across all namespaces:
43+
```yaml
44+
apiVersion: projectcalico.org/v3
45+
kind: GlobalNetworkPolicy
46+
metadata:
47+
name: allow-stellar-p2p
48+
spec:
49+
selector: app.kubernetes.io/name == 'stellar-node'
50+
types:
51+
- Ingress
52+
- Egress
53+
ingress:
54+
- action: Allow
55+
protocol: TCP
56+
destination:
57+
ports: [11625]
58+
egress:
59+
- action: Allow
60+
protocol: TCP
61+
destination:
62+
ports: [11625]
63+
```
64+
65+
### 2.2 Cilium CNI
66+
Cilium uses eBPF (Extended Berkeley Packet Filter) to route and secure network packets directly in the Linux kernel without iptables overhead.
67+
- **eBPF-based Host Routing**: Enables lower latency and higher throughput, crucial for Validator synchronization.
68+
- **CiliumNetworkPolicy**: Standard policy limiting ingress to authorized endpoints:
69+
```yaml
70+
apiVersion: "cilium.io/v2"
71+
kind: CiliumNetworkPolicy
72+
metadata:
73+
name: secure-validator-p2p
74+
namespace: stellar
75+
spec:
76+
endpointSelector:
77+
matchLabels:
78+
app.kubernetes.io/component: stellar-validator
79+
ingress:
80+
- fromEndpoints:
81+
- matchLabels:
82+
app.kubernetes.io/component: stellar-validator
83+
toPorts:
84+
- ports:
85+
- port: "11625"
86+
protocol: TCP
87+
```
88+
89+
---
90+
91+
## 3. BGP Configuration for Multi-Cluster Networking
92+
93+
BGP (Border Gateway Protocol) allows the Kubernetes cluster nodes to advertise Pod and Service IP blocks directly to the physical network routers.
94+
95+
### 3.1 Calico BGP Configuration
96+
To configure BGP peering with external top-of-rack (ToR) switches:
97+
```yaml
98+
apiVersion: projectcalico.org/v3
99+
kind: BGPPeer
100+
metadata:
101+
name: tor-switch-peer
102+
spec:
103+
peerIP: 192.168.1.1
104+
asNumber: 65001
105+
```
106+
107+
### 3.2 MetalLB BGP Mode Configuration
108+
MetalLB implements load balancers in bare-metal clusters. In BGP mode, MetalLB establishes a BGP session with the router to advertise the LoadBalancer IP.
109+
```yaml
110+
apiVersion: metallb.io/v1beta2
111+
kind: BGPPeer
112+
metadata:
113+
name: core-router
114+
namespace: metallb-system
115+
spec:
116+
peerAddress: 10.0.0.1
117+
peerASN: 64512
118+
myASN: 64513
119+
---
120+
apiVersion: metallb.io/v1beta1
121+
kind: IPAddressPool
122+
metadata:
123+
name: stellar-ips
124+
namespace: metallb-system
125+
spec:
126+
addresses:
127+
- 192.168.10.100-192.168.10.120
128+
---
129+
apiVersion: metallb.io/v1beta1
130+
kind: BGPAdvertisement
131+
metadata:
132+
name: advertise-stellar-ips
133+
namespace: metallb-system
134+
spec:
135+
ipAddressPools:
136+
- stellar-ips
137+
```
138+
139+
---
140+
141+
## 4. Load Balancer Integration
142+
143+
### 4.1 MetalLB (Bare-Metal)
144+
- Set up MetalLB in either Layer 2 mode (ARP-based) or BGP mode as shown above.
145+
- In Layer 2 mode, ensure that `kube-proxy` config has `strictARP: true` enabled.
146+
147+
### 4.2 Cloud Provider Load Balancers
148+
For AWS deployments, use the AWS Load Balancer Controller to provision Network Load Balancers (NLBs) for low-latency TCP routing:
149+
```yaml
150+
apiVersion: v1
151+
kind: Service
152+
metadata:
153+
name: validator-p2p
154+
namespace: stellar
155+
annotations:
156+
service.beta.kubernetes.io/aws-load-balancer-type: "external"
157+
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
158+
service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
159+
spec:
160+
type: LoadBalancer
161+
selector:
162+
app.kubernetes.io/name: stellar-node
163+
ports:
164+
- port: 11625
165+
targetPort: 11625
166+
protocol: TCP
167+
```
168+
169+
---
170+
171+
## 5. Mutual TLS (mTLS) and Service Mesh
172+
173+
Integrating a Service Mesh secures inter-pod communication through mutual TLS (mTLS).
174+
175+
### 5.1 Istio Service Mesh
176+
1. **Enable Sidecar Injection**: Label the namespace to inject Envoy proxies automatically:
177+
```bash
178+
kubectl label namespace stellar istio-injection=enabled
179+
```
180+
2. **Enforce Strict mTLS**:
181+
```yaml
182+
apiVersion: security.istio.io/v1beta1
183+
kind: PeerAuthentication
184+
metadata:
185+
name: default
186+
namespace: stellar
187+
spec:
188+
mtls:
189+
mode: STRICT
190+
```
191+
192+
### 5.2 Linkerd Service Mesh
193+
1. Inject the Linkerd proxy by adding the annotation to your `StellarNode` spec metadata:
194+
```yaml
195+
spec:
196+
metadata:
197+
annotations:
198+
linkerd.io/inject: enabled
199+
```
200+
201+
---
202+
203+
## 6. Network Performance Tuning
204+
205+
To optimize network throughput and reduce latency for high-speed blockchain state sync:
206+
1. **TCP Socket Buffers**: Increase sysctl socket memory allocation limits on the host nodes:
207+
```bash
208+
sysctl -w net.core.rmem_max=16777216
209+
sysctl -w net.core.wmem_max=16777216
210+
```
211+
2. **Cilium eBPF Host Routing**: Skip standard iptables connection tracking overhead using Cilium's direct routing model:
212+
```bash
213+
helm upgrade cilium cilium/cilium --set bpf.masquerade=true --set hostServices.enabled=true
214+
```
215+
216+
---
217+
218+
## 7. Troubleshooting and Common Issues
219+
220+
Refer to the [Networking Troubleshooting Guide](../troubleshooting/networking.md) for step-by-step diagnostic actions for:
221+
- `Connection Refused`
222+
- `No Route to Host`
223+
- DNS Resolution Failures
224+
- CNI Status checks
225+
- mTLS handshake errors
226+
227+
---
228+
229+
## 8. Network Monitoring & Metrics
230+
231+
### 8.1 Prometheus ServiceMonitor
232+
Create a ServiceMonitor to collect network performance metrics from the nodes:
233+
```yaml
234+
apiVersion: monitoring.coreos.com/v1
235+
kind: ServiceMonitor
236+
metadata:
237+
name: stellar-node-monitor
238+
namespace: stellar
239+
spec:
240+
selector:
241+
matchLabels:
242+
app.kubernetes.io/name: stellar-node
243+
endpoints:
244+
- port: metrics
245+
interval: 10s
246+
```
247+
248+
### 8.2 Recommended Grafana Panels
249+
- **Active Connections**: Track the total number of connected P2P peers.
250+
- **Network I/O Bytes**: Inbound and outbound bandwidth utilization.
251+
- **Packet Retransmission Rate**: High values indicate packet loss and potential CNI/network congestion.

0 commit comments

Comments
 (0)