|
| 1 | +--- |
| 2 | +title: Networking & Transport Non-Negotiables |
| 3 | +description: File-scoped non-negotiables for transport-layer code and protobuf schemas - HTTP/gRPC client reuse, TCP keepalive, wire-format choice, protobuf field-number safety, idle-timeout discipline. |
| 4 | +priority: 325 |
| 5 | +alwaysApply: false |
| 6 | +files: |
| 7 | + include: |
| 8 | + - "**/*.proto" |
| 9 | + - "**/buf.yaml" |
| 10 | + - "**/buf.gen.yaml" |
| 11 | + - "**/buf.lock" |
| 12 | + - "**/buf.work.yaml" |
| 13 | + - "**/.connect.yaml" |
| 14 | + - "**/grpc.config.yaml" |
| 15 | +--- |
| 16 | + |
| 17 | +# Networking & Transport Non-Negotiables |
| 18 | + |
| 19 | +**Audience:** engineers editing protobuf schemas, gRPC client/server code, network configuration (load balancers, edge), or HTTP client wiring. |
| 20 | + |
| 21 | +**Scope:** the rules that auto-load when you touch those files. The depth and reasoning live in the `networking-transport` skill (`skills/networking-transport/SKILL.md` plus references for the gRPC-vs-REST decision matrix and protobuf schema evolution). This rule codifies the **non-negotiables** so they appear in context without needing to invoke the skill. |
| 22 | + |
| 23 | +> [!IMPORTANT] |
| 24 | +> This rule complements `320-api-design.mdc` (API surface: verbs, status codes, versioning) and `316-zero-trust.mdc` Section 2 (network-tier security on top of the transport). Use this rule for transport-and-wire choices; use those for the API surface and the auth model. |
| 25 | + |
| 26 | +--- |
| 27 | + |
| 28 | +## Transport non-negotiables |
| 29 | + |
| 30 | +### NN1. HTTP and gRPC clients are reused across requests |
| 31 | + |
| 32 | +A fresh client per request defeats connection pooling, defeats TCP keepalive, and burns 50-150ms / call on TCP+TLS handshake. The single most common production performance bug. |
| 33 | + |
| 34 | +- **Go:** package-level `http.Client` with a tuned `Transport`; do not construct one per call. |
| 35 | +- **Python:** module-level `requests.Session()` (or `httpx.Client`) with a mounted `HTTPAdapter`; do not call `requests.get()` ad-hoc in hot paths. |
| 36 | +- **Node:** shared `https.Agent({ keepAlive: true })` passed via `fetch(url, { agent })` (Node 18 and older defaults to `keepAlive: false`). |
| 37 | +- **gRPC:** one `Channel` per `(target, credentials)` pair, reused for the process lifetime; never `new Channel()` per call. |
| 38 | + |
| 39 | +### NN2. TCP keepalive is set explicitly and tuned below the load balancer idle timeout |
| 40 | + |
| 41 | +Linux defaults (`tcp_keepalive_time = 7200s` = 2 hours) are wrong for cloud. Load balancers cut idle connections much sooner; the client never notices until the next send fails. |
| 42 | + |
| 43 | +Set keepalive interval **less than** the proxy's idle timeout: |
| 44 | + |
| 45 | +| Proxy | Idle timeout | Suggested keepalive | |
| 46 | +|---|---|---| |
| 47 | +| AWS Network Load Balancer (NLB) | 350s (fixed) | 60-120s | |
| 48 | +| AWS Application LB (ALB) | 60s default (1-4000s configurable) | 30s (or half the configured) | |
| 49 | +| AWS API Gateway REST | 29s (fixed) | Don't keepalive; reconnect per request | |
| 50 | +| Cloudflare WebSocket | ~100s free / ~300s paid | Application-level heartbeat every 30-60s | |
| 51 | +| Nginx (`proxy_read_timeout`) | 60s | 30s | |
| 52 | +| Envoy (`stream_idle_timeout`) | 1h | 30-300s by use case | |
| 53 | + |
| 54 | +### NN3. Wire format choice is by callers and use case, not by hype |
| 55 | + |
| 56 | +| Caller / use case | Default choice | Why | |
| 57 | +|---|---|---| |
| 58 | +| External public API, browser-callable, third-party developers | **REST + JSON** | Curl-debuggable, OpenAPI tooling, broadest support, no code-gen burden on callers | |
| 59 | +| Internal east-west, polyglot, you own both ends | **gRPC + Protobuf** | Schema-first, code-gen, streaming, ~5-10x smaller wire, ~3-5x faster ser/deser | |
| 60 | +| Browser <-> backend with Protobuf contract | **Connect** (or REST) | gRPC needs gRPC-Web + proxy; Connect calls work with plain `fetch` | |
| 61 | +| Long-lived bidirectional service-to-service streaming | **gRPC streaming** | First-class concept; one HTTP/2 stream per session | |
| 62 | +| Client-driven shape (mobile picks fields per query) | **GraphQL** | Single endpoint, projection-by-query | |
| 63 | + |
| 64 | +Do not ship gRPC as a **public** API unless your callers have specifically agreed to the tooling burden. The browser-friendly, curl-debuggable, cache-friendly default is REST. |
| 65 | + |
| 66 | +### NN4. Long-lived connections have application-level heartbeats |
| 67 | + |
| 68 | +WebSocket, SSE, and gRPC streaming all die silently when an intermediate proxy hits its idle timeout. TCP keepalive alone is not enough; many proxies measure idle at the application layer. |
| 69 | + |
| 70 | +- WebSocket: ping/pong every 30-60s (under the smallest proxy idle in the path). |
| 71 | +- SSE: emit a `: keepalive` comment line every 30-60s. |
| 72 | +- gRPC streaming: rely on HTTP/2 PING (configured via channel keepalive options) plus application-level heartbeats for app-aware liveness. |
| 73 | +- All: implement reconnect with exponential backoff + jitter; do not assume one connection lasts forever. |
| 74 | + |
| 75 | +### NN5. HTTP/3 is enabled at the edge for mobile / cross-region / lossy paths (where supported) |
| 76 | + |
| 77 | +HTTP/2's TCP-level head-of-line blocking degrades P95/P99 on lossy networks. HTTP/3 over QUIC eliminates it at the transport layer. |
| 78 | + |
| 79 | +- Public-facing edge (Cloudflare, Fastly, CloudFront): HTTP/3 on if your CDN supports it (all major CDNs do as of 2025+). |
| 80 | +- Mobile-heavy traffic: HTTP/3 helps disproportionately. |
| 81 | +- Internal east-west on LAN with <0.001% loss: HTTP/2 is fine; HTTP/3 buys little. |
| 82 | +- Prerequisites: TLS 1.3, UDP/443 reachable end-to-end, `alt-svc` header advertised. Some corporate proxies filter UDP; provide HTTP/2 fallback. |
| 83 | + |
| 84 | +--- |
| 85 | + |
| 86 | +## Protobuf non-negotiables |
| 87 | + |
| 88 | +When editing any `.proto` file: |
| 89 | + |
| 90 | +### PB1. Never re-number a field |
| 91 | + |
| 92 | +Field numbers are the wire identity. Renumbering corrupts every existing serialized message and breaks every downstream consumer. There is **no** safe way to renumber a field that has shipped. |
| 93 | + |
| 94 | +If you must change a field's semantics, add a new field with a new number and `reserve` the old one. |
| 95 | + |
| 96 | +### PB2. Removed fields get `reserved` for both number AND name |
| 97 | + |
| 98 | +Reserving the number prevents future re-use that would corrupt old data. Reserving the name prevents accidental re-introduction of the same logical field with different semantics. |
| 99 | + |
| 100 | +```protobuf |
| 101 | +message User { |
| 102 | + string id = 1; |
| 103 | + string email = 2; |
| 104 | + // string display_name = 3; // removed in v1.4 |
| 105 | + reserved 3; |
| 106 | + reserved "display_name"; |
| 107 | +} |
| 108 | +``` |
| 109 | + |
| 110 | +### PB3. Use `optional` keyword or wrapper types for proto3 presence semantics |
| 111 | + |
| 112 | +In proto3, scalar fields with their default value (zero, empty string, false) do not appear on the wire. Receivers cannot distinguish "set to 0" from "absent". When that distinction matters: |
| 113 | + |
| 114 | +```protobuf |
| 115 | +// preferred (proto3 since release 3.15, Feb 2020) |
| 116 | +optional int32 max_attempts = 1; |
| 117 | + |
| 118 | +// fallback for older toolchains |
| 119 | +import "google/protobuf/wrappers.proto"; |
| 120 | +google.protobuf.Int32Value max_attempts = 1; |
| 121 | +``` |
| 122 | + |
| 123 | +Do not invent sentinel values (`-1` means absent) - brittle, undocumented, easy to miss. |
| 124 | + |
| 125 | +### PB4. Package versioning convention: `<org>.<domain>.v<N>` |
| 126 | + |
| 127 | +```protobuf |
| 128 | +package myorg.users.v1; |
| 129 | +``` |
| 130 | + |
| 131 | +Major-version the package, not individual messages. `v1` stays for backward compatibility; breaking changes go in `v2`. Service-versioning (`UsersV1`, `UsersV2`) is acceptable as an alternative if used consistently. |
| 132 | + |
| 133 | +### PB5. `buf breaking` runs in CI against the merge base |
| 134 | + |
| 135 | +The single highest-value gate for protobuf schema discipline. Catches every PB1-PB3 violation automatically. |
| 136 | + |
| 137 | +```yaml |
| 138 | +# .buf.yaml |
| 139 | +version: v1 |
| 140 | +breaking: |
| 141 | + use: |
| 142 | + - WIRE_JSON |
| 143 | +``` |
| 144 | + |
| 145 | +```yaml |
| 146 | +# .github/workflows/proto.yml step |
| 147 | +- run: buf breaking --against '.git#branch=main' |
| 148 | +``` |
| 149 | + |
| 150 | +Also recommended: `buf lint` for style, `buf format` for normalization, `protovalidate` for cross-language validation rules expressed in the `.proto` itself. |
| 151 | + |
| 152 | +--- |
| 153 | + |
| 154 | +## Common anti-patterns (reviewers reject these) |
| 155 | + |
| 156 | +| Anti-pattern | Why it's wrong | |
| 157 | +|---|---| |
| 158 | +| `http.Get(url)` / `requests.get(url)` / `fetch(url)` in hot paths | New client per call -> no keepalive -> handshake tax per request | |
| 159 | +| Default OS TCP keepalive in cloud | 2-hour idle means dead-peer detection happens long after the LB cut | |
| 160 | +| `keepalive > load balancer idle timeout` | LB wins; "random" disconnects appear in P99 | |
| 161 | +| Renumbering a protobuf field | Wire-incompatible; corrupts every existing serialized message | |
| 162 | +| Removing a protobuf field without `reserved` | Sets up the next change to silently reuse the number and corrupt data | |
| 163 | +| Scalar field for true optional semantics in proto3 | Default-value doesn't serialize; receiver can't distinguish set-to-zero from absent | |
| 164 | +| gRPC as a browser-callable public API without Connect / gRPC-Web | Doesn't work; teams discover too late | |
| 165 | +| Stripping unknown protobuf fields in middleware | Destroys forward compatibility | |
| 166 | +| One protobuf file with everything | Hard to evolve; circular imports | |
| 167 | +| `rev: nightly` or `rev: main` on protobuf-related pre-commit hooks | Mutable references; pin to a release tag | |
| 168 | + |
| 169 | +--- |
| 170 | + |
| 171 | +## Reviewer checklist |
| 172 | + |
| 173 | +For HTTP / gRPC client code: |
| 174 | + |
| 175 | +- [ ] Client / channel created once (package-level, DI, singleton) and reused |
| 176 | +- [ ] Explicit timeout set (overall request, not just connect) |
| 177 | +- [ ] TCP keepalive interval set, less than load balancer idle timeout |
| 178 | +- [ ] Pool size (`MaxIdleConnsPerHost` etc.) sized to workload, not left at default |
| 179 | +- [ ] TLS minimum is 1.2; prefer 1.3 |
| 180 | +- [ ] Retry policy bounded (max attempts, max total time, exponential backoff with jitter) |
| 181 | +- [ ] Metrics emitted: latency histogram, status counter, connection-reuse ratio |
| 182 | + |
| 183 | +For `.proto` changes: |
| 184 | + |
| 185 | +- [ ] No field renumbered (verified by `buf breaking`) |
| 186 | +- [ ] Removed fields have `reserved` for both number AND name |
| 187 | +- [ ] New fields use unused numbers (never reused) |
| 188 | +- [ ] Optional semantics use `optional` keyword or wrapper types, not scalar defaults |
| 189 | +- [ ] Package is version-suffixed (`org.domain.vN`) |
| 190 | +- [ ] `buf breaking` runs in CI against the merge base |
| 191 | +- [ ] Generated code policy is consistent (committed XOR generated in CI; not both) |
| 192 | + |
| 193 | +For long-lived connections (WebSocket / SSE / gRPC streaming): |
| 194 | + |
| 195 | +- [ ] Application-level heartbeat every 30-60s |
| 196 | +- [ ] Reconnect logic with exponential backoff + jitter |
| 197 | +- [ ] Idle timeout configured at every proxy in the path (or smallest-wins documented) |
| 198 | + |
| 199 | +--- |
| 200 | + |
| 201 | +## See also |
| 202 | + |
| 203 | +- Skill: `networking-transport` - depth on TCP keepalive, HoL blocking, TTFB budget, HTTP/n choice, wire format decision, long-lived connections, edge/CDN, observability |
| 204 | +- Skill: `networking-transport/references/grpc-vs-rest-decision.md` - decision matrix with worked examples |
| 205 | +- Skill: `networking-transport/references/protobuf-schema-evolution.md` - schema evolution rules in depth |
| 206 | +- Rule: `320-api-design.mdc` - API surface (verbs, status codes, versioning) |
| 207 | +- Rule: `316-zero-trust.mdc` - Network section (mTLS, default-deny egress) |
| 208 | +- Rule: `330-observability.mdc` - logging, metrics, tracing patterns |
| 209 | +- Rule: `400-cloudflare.mdc` - edge / CDN patterns |
| 210 | +- Rule: `410-aws.mdc` - AWS LB and API Gateway idle-timeout specifics |
| 211 | +- Rule: `483-kafka.mdc` - Kafka transport + Avro/Protobuf schema registry |
0 commit comments