Skip to content

v1.5.6

Choose a tag to compare

@Romain-Grosos Romain-Grosos released this 10 May 20:23
984fd3f

v1.5.6 - ConsistentHash load balancing wired

A small reliability patch release. One bug fix, no new feature, no schema migration, no config change. Upgrade is a hot drop-in.

Fixed

  • ConsistentHash load-balancing strategy was a silent no-op: a route configured with load_balancing = "consistent_hash" behaved exactly like round_robin, the operator's intent for client-IP-based session stickiness was discarded server-side. Root cause: the selection match in proxy_wiring::upstream_peer (lorica/src/proxy_wiring.rs) handled PeakEwma, Random, LeastConn explicitly, then routed both RoundRobin and ConsistentHash through the same _ => catch-all that calls the smooth weighted round-robin selector. The original comment on that arm even acknowledged the gap ("covers RoundRobin and ConsistentHash") but no one wired the consistent path. Production shape: an operator who flipped a route to consistent_hash to keep a given client IP pinned to one backend (typical use case: sticky cache or stateful upstream that does not share state across replicas) saw the request stream spread across all healthy backends like before, with no log line indicating the strategy had been silently downgraded. The dashboard, API, and config schema all accepted the value, so the only way to detect the bug was to compare actual upstream selection against the expected hash.

    Fix: a dedicated LoadBalancing::ConsistentHash arm now feeds ctx.client_ip (already populated in request_filter, honouring X-Forwarded-For when trusted_proxies says the immediate peer is trusted) into the Ketama hash ring (lorica-ketama::Continuum). The ring is built from each healthy backend's (address, weight), weight maps to ketama's points-per-bucket = weight * 160 so a backend with weight=2 gets twice the slice of the keyspace. Backend addresses that do not parse as SocketAddr (Unix-domain backends prefixed with unix:) are skipped at ring construction; if every healthy backend is UDS, the selection falls back to smooth weighted round-robin for that request rather than erroring 502. Same fallback applies when ctx.client_ip is None. Sticky-session cookies still take precedence: the existing sticky_backend_idx short-circuit fires before the load-balancing match, so a route with both sticky_session = true AND load_balancing = "consistent_hash" honours the cookie first and only falls through to the hash on the first request.

    Selection is extracted to proxy_wiring::helpers::pick_consistent_hash(&[(addr, weight)], client_ip) so the pure path is unit-tested in isolation. Tests: 5 new in proxy_wiring::tests (same-client stickiness, distinct-clients spread, None/empty client IP falls back via caller, UDS-only pool falls back via caller, stable selection when an unselected backend is removed - the consistent-hash invariant vs hash-modulo-N).

    Known limitations carried as backlog for v1.6.0: (a) the ring is rebuilt per request (O(n * 160 * weight); on a route with 5 backends and weight=1 each this is ~3 us measured cold, acceptable for a hotfix but a fixed cost worth caching against the route entry like wrr_state), (b) UDS backends are silently excluded from the hash key space, matching the upstream lorica-lb/src/selection/consistent.rs behaviour - making UDS hashable needs a different ketama variant.

Upgrade notes

  • No schema migration, no config change.
  • No operator action required for routes that did NOT use consistent_hash: behaviour is unchanged.
  • Operator action recommended for any route that was configured with load_balancing = "consistent_hash": before v1.5.6, that route was effectively running round_robin. After upgrade it will start hashing on client IP, so the load distribution will change shape (the per-IP stickiness the operator originally asked for). If the upstream is stateless or session state is shared across backends, no further action; if the upstream depends on stickiness (cache locality, in-memory session, etc.) the v1.5.6 behaviour is the correct one.
  • A v1.5.5 -> v1.5.6 upgrade is a hot drop-in: stop the service, replace the binary / .deb / .rpm, restart. The SQLite store, certificates, configs, sessions all carry forward unchanged.

Artifacts

  • .deb (Ubuntu / Debian, amd64): built via dist/build-deb.sh
  • .rpm (RHEL / Rocky / Alma, x86_64): built via dist/rpm/lorica.spec
  • Source tarball: auto-generated by GitHub from the tag

Verification (operator-side after deploy)

  1. Identify a route configured with load_balancing = "consistent_hash" (or temporarily configure one for the test with two backends).
  2. From a client at a fixed IP, send 10 requests to the route. All 10 must hit the same backend (visible in the access log per-backend counter, or lorica_backend_requests_total Prometheus counter).
  3. Repeat from a second client IP. The second client should hit a backend, possibly the same possibly a different one depending on the hash; what matters is that it stays stable across its own 10 requests.
  4. Confirm previously-working round_robin / peak_ewma / least_conn / random routes still behave as before.

Full changelog

Compare v1.5.5...v1.5.6