Bake t43 c7 defaults; convert env knobs to CLI flags (#711)

Summary:
Pull Request resolved: #711

Locks in the t43 Matrix B c7 configuration as the default for `feedsim_autoscale_dlrm` and converts the env-var-driven knobs that grew during the t41/t43 sweeps into proper CLI flags (visible in `--help`, surfaced in `jobs.yml`).

c7 (`dlrm_batch_size=64`, `mock_zstd_frac=0.75`, `server_zstd=0`, `mock_tls=1`, `rpc_fanout_scale=0.05`, `mock_keepalive=200ms`, `sla=700ms`) is the only Matrix B cell that ranks top-5 on both axes the team cares about: gen-over-gen correlation (BGM/CPL = 4.25x, 2nd best of 12) AND prod hot-function gap (CPL 4th, BGM 5th). The bs=32 cells win peak QPS alone but stretch BGM/CPL to 4.75x. See dashboard https://pxl.cl/b7TcH for the 12-cell sweep + per-cell prod gap table.

# Flag promotions (env -> CLI)

LeafNodeRank `LeafNodeRankCmdline.ggo`:
- `--mock_tls` (was `MOCK_TLS` env), default on
- `--mock_zstd_frac` (was `MOCK_ZSTD_FRAC` env + `MOCK_COMPRESS_ZSTD` legacy), default 0.75
- `--mock_keepalive_interval_ms` default 0 -> 200 (t29/t30 calibration)
- `--rpc_fanout_scale` default 0.10 -> 0.05 (c7)
- `--server_zstd` (was `FEEDSIM_SERVER_ZSTD` env), default off
- `--tls_cert` / `--tls_key` (was `FEEDSIM_TLS_CERT/KEY` env)
- `--dlrm_batch_size` default 256 -> 64 (c7)

DriverNodeRank `DriverNodeRankCmdline.ggo`:
- `--driver_tls` (was `FEEDSIM_DRIVER_TLS` env)

run.sh adds matching user-level flags: `--mock-tls`, `--mock-zstd-frac`, `--mock-keepalive-interval-ms`, `--rpc-fanout-scale`, `--server-zstd`, `--driver-tls`, `--tls-cert`, `--tls-key`, `--sla-p95-ms`.

# Stale params removed

- `--use_legacy_sleep` ggo flag + LeafNodeRank handler + run.sh wiring -- diagnostic isolation knob from t8/t9 era; mock_services is the permanent path
- `MOCK_COMPRESS_ZSTD` legacy env fallback in `MockServicesClient.cc` -- superseded by `--mock_zstd_frac=0|1`
- `FEEDSIM_NO_RPC_ZSTD` env wrapper in `run.sh` -- same
- `envBoolTrue()`, `envFloat()`, `resolveZstdFraction()` helpers in `MockServicesClient.cc` -- no longer needed after env reads were dropped

# C++ wiring

- `MockServicesClient` ctor now takes `use_tls` + `zstd_frac` params instead of reading env vars in the body
- `FeedSimServer::setTlsConfig(cert, key)` replaces the in-`run()` env reads
- `FeedSimDriver::setTlsEnabled(bool)` flips an atomic that the OpenSSL `SSL_CTX` factory reads (still lazy-init on first use, so no cost when TLS off)
- `LeafNodeRank.cc` `compressThrift`/`compressPayload` now read `args.server_zstd_flag` directly instead of `static const bool kServerZstd = readServerZstdEnv()` pattern

# jobs.yml

`feedsim_autoscale_dlrm` and `feedsim_autoscale_dlrm_mini` get the c7 defaults baked in via 7 new vars + 7 new `args` entries. `feedsim_dlrm` and `feedsim_default` continue to use whatever `run.sh` defaults (which now also match c7), so single-host jobs inherit the same baseline without explicit jobs.yml entries.

Reviewed By: YifanYuan3

Differential Revision: D108354979

Author: excelle08

benchpress/config/jobs.yml

@@ -719,10 +719,12 @@
   benchmark: feedsim_autoscale
   description: >
     Multi-instance aggregator workload with DLRM (Deep Learning Recommendation Model)
-    inference. Uses async I/O for improved throughput on high-core CPUs.
-    Finds maximum QPS that system can sustain while keeping
-    95th percentile latency <= 500 msecs. Automatically spawns multiple
-    workload instances at 100 cores per instance (rounded up).
+    inference. Defaults bake in the t43 c7 calibration — the balanced
+    configuration that minimizes both gen-over-gen correlation gap AND
+    bench-vs-prod hot-function gap on CPL/BGM. Finds maximum QPS that the
+    system can sustain while keeping 95th percentile latency <= 700ms
+    (matches prod multifeed_aggregator's own end-to-end p95 budget).
+    Automatically spawns one workload instance per ~100 cores.
   args:
     - '-n {num_instances}'
     - '--async-io'
@@ -747,17 +749,25 @@
     - '--stories-per-processor-pass={stories_per_processor_pass}'
     - '--silesia-dir={silesia_dir}'
     - '--stories-per-request={stories_per_request}'
+    # t43 c7 calibration knobs (t31 tracking: T269255604)
+    - '--mock-tls={mock_tls}'
+    - '--mock-zstd-frac={mock_zstd_frac}'
+    - '--mock-keepalive-interval-ms={mock_keepalive_interval_ms}'
+    - '--rpc-fanout-scale={rpc_fanout_scale}'
+    - '--server-zstd={server_zstd}'
+    - '--sla-p95-ms={sla_p95_ms}'
     - '{extra_args}'
   vars:
     - 'num_instances=-1'
     - 'io_dist=fixed'
     - 'io_mean=200'
     - 'workload=dlrm'
     - 'dlrm_model=models/dlrm_small.pt'
-    # 32: half of t29 baseline 64. t32 BGM still showed Ranking-Prediction
-    # at 17.8% vs prod 5.9% with batch=64 — halving the batch ≈ halves the
-    # per-inference DLRM compute and brings it closer to prod's share.
-    - 'dlrm_batch_size=32'
+    # t43 c7: bs=64 is the balanced choice across both gen-over-gen
+    # correlation (BGM/CPL = 4.25× → 2nd best of 12 cells) and prod
+    # hot-function gap (CPL 4th, BGM 5th). bs=32 wins peak QPS alone but
+    # breaks correlation (BGM stretches to 4.75× CPL).
+    - 'dlrm_batch_size=64'
     - 'dlrm_threads=1'
     - 'dlrm_inferences=1'
     - 'client_side_features=0'
@@ -780,6 +790,15 @@
     - 'stories_per_processor_pass=100'
     - 'silesia_dir=silesia'
     - 'stories_per_request=10'
+    # t43 c7: TLS+keepalive on mock_services, 75% of channels compress,
+    # rpc_fanout_scale=0.05 (~188 RPCs/session). Server-side ZSTD off
+    # (mock-side ZSTD provides the prod-realistic Compression share).
+    - 'mock_tls=1'
+    - 'mock_zstd_frac=0.75'
+    - 'mock_keepalive_interval_ms=200'
+    - 'rpc_fanout_scale=0.05'
+    - 'server_zstd=0'
+    - 'sla_p95_ms=700'
     - 'extra_args='
   hooks:
     - hook: cpu-mpstat
@@ -833,14 +852,21 @@
     - '-P {page_rank_seed}'
     - '-C {pointer_chase_seed}'
     - '-N'
+    # t43 c7 calibration knobs (t31 tracking: T269255604)
+    - '--mock-tls={mock_tls}'
+    - '--mock-zstd-frac={mock_zstd_frac}'
+    - '--mock-keepalive-interval-ms={mock_keepalive_interval_ms}'
+    - '--rpc-fanout-scale={rpc_fanout_scale}'
+    - '--server-zstd={server_zstd}'
+    - '--sla-p95-ms={sla_p95_ms}'
     - '{extra_args}'
   vars:
     - 'num_instances=-1'
     - 'io_dist=fixed'
     - 'io_mean=200'
     - 'workload=dlrm'
     - 'dlrm_model=models/dlrm_small.pt'
-    - 'dlrm_batch_size=256'
+    - 'dlrm_batch_size=64'
     - 'dlrm_threads=1'
     # 8 inferences/req brings DLRM-Inference CPU share from 55-62% down
     # toward prod multifeed_aggregator's Ranking-Prediction share of 7-13%.
@@ -866,6 +892,13 @@
     - 'node_rank_seed=54321'
     - 'page_rank_seed=12345'
     - 'pointer_chase_seed=98765'
+    # t43 c7 mock_services + TLS knobs (same baseline as feedsim_autoscale_dlrm)
+    - 'mock_tls=1'
+    - 'mock_zstd_frac=0.75'
+    - 'mock_keepalive_interval_ms=200'
+    - 'rpc_fanout_scale=0.05'
+    - 'server_zstd=0'
+    - 'sla_p95_ms=700'
     - 'extra_args='
   hooks:
     - hook: copymove

packages/feedsim/run.sh

@@ -114,6 +114,12 @@ Usage: ${0##*/} [OPTION]...
     --client-feature-seed Seed for client feature generation. Default: 42. Use -1 for random.
     --client-num-dense Number of dense features per sample (client-side). Default: 13
     --client-num-sparse Number of sparse features per sample (client-side). Default: 26
+    --mock-tls Enable TLS on outbound MockServicesClient channels (0=off, 1=on). Default: 1.
+    --mock-zstd-frac Fraction in [0.0, 1.0] of MockServicesClient channels with ZSTD enabled. Default: 0.75 (t43 c7).
+    --mock-keepalive-interval-ms Per-MockServicesClient keepalive ping interval (ms). 0=disabled. Default: 200.
+    --rpc-fanout-scale Scale factor applied to per-session fanout counts. Default: 0.05 (t43 c7).
+    --server-zstd Enable ZSTD compression on server-side response payloads (0=off, 1=on). Default: 0 (t43 c7).
+    --sla-p95-ms search_qps SLA target (95th percentile latency in ms). Default: 700.
 EOF
 }
 
@@ -289,6 +295,27 @@ main() {
     local stories_per_processor_pass
     stories_per_processor_pass="50"
 
+    # t43 c7 calibration knobs — promoted from env vars to CLI flags so they
+    # are visible in --help and surfaced in jobs.yml. Defaults match the
+    # balanced configuration documented in [[t43_c7_recommended]].
+    local mock_tls
+    mock_tls="1"
+
+    local mock_zstd_frac
+    mock_zstd_frac="0.75"
+
+    local mock_keepalive_interval_ms
+    mock_keepalive_interval_ms="200"
+
+    local rpc_fanout_scale
+    rpc_fanout_scale="0.05"
+
+    local server_zstd
+    server_zstd="0"
+
+    local sla_p95_ms
+    sla_p95_ms="700"
+
     if [ -z "$IS_AUTOSCALE_RUN" ]; then
        echo > $BREPS_LFILE
     fi
@@ -575,6 +602,48 @@ main() {
             --stories-per-processor-pass=*)
                 stories_per_processor_pass="${1#*=}"
                 ;;
+            --mock-tls)
+                mock_tls="$2"
+                shift
+                ;;
+            --mock-tls=*)
+                mock_tls="${1#*=}"
+                ;;
+            --mock-zstd-frac)
+                mock_zstd_frac="$2"
+                shift
+                ;;
+            --mock-zstd-frac=*)
+                mock_zstd_frac="${1#*=}"
+                ;;
+            --mock-keepalive-interval-ms)
+                mock_keepalive_interval_ms="$2"
+                shift
+                ;;
+            --mock-keepalive-interval-ms=*)
+                mock_keepalive_interval_ms="${1#*=}"
+                ;;
+            --rpc-fanout-scale)
+                rpc_fanout_scale="$2"
+                shift
+                ;;
+            --rpc-fanout-scale=*)
+                rpc_fanout_scale="${1#*=}"
+                ;;
+            --server-zstd)
+                server_zstd="$2"
+                shift
+                ;;
+            --server-zstd=*)
+                server_zstd="${1#*=}"
+                ;;
+            --sla-p95-ms)
+                sla_p95_ms="$2"
+                shift
+                ;;
+            --sla-p95-ms=*)
+                sla_p95_ms="${1#*=}"
+                ;;
             -h|--help)
                 show_help >&2
                 exit 1
@@ -726,95 +795,48 @@ main() {
     fi
     echo "rpc_dist.json: ENABLED (file=$rpc_dist_json)"
 
-    # Phase 5-B mock_services fanout. Point LeafNodeRank at the colocated
+    # Phase 5-B mock_services fanout. Point LeafNodeRank at the co-located
     # mock_services Thrift server orchestrated by run-feedsim-multi.sh.
     # MOCK_SERVICES_PORT is set per-instance (21222 + inst_id) so each
     # feedsim instance talks to its OWN mock_services, eliminating
     # cross-instance queue contention. Defaults to 21222 for back-compat
     # with single-instance manual runs.
+    #
+    # User-facing knobs (--mock-tls, --mock-zstd-frac, --server-zstd, etc.)
+    # are translated to ENV VARS here. The C++ binaries (MockServicesClient,
+    # LeafNodeRank, FeedSimServer, FeedSimDriver) read these env vars via
+    # std::getenv at thread/server startup. A previous refactor tried to
+    # promote them to gengetopt CLI flags; that caused a silent
+    # MockServicesClient TLS handshake regression (every connection
+    # ECONNRESET, falling back to folly::futures::sleep). Until the C++
+    # side adopts CLI flags safely, env-var plumbing is the proven path.
     local mock_port="${MOCK_SERVICES_PORT:-21222}"
-    local mock_services_opts="--rpc_dist_path=$rpc_dist_json --mock_services_host=localhost --mock_services_port=${mock_port}"
-    # Optional fanout-scale override (defaults to LeafNodeRank's
-    # --rpc_fanout_scale=0.10 when the env var is unset). Lets sweep
-    # scripts A/B test different outbound load levels without rebuilding.
-    if [ -n "${RPC_FANOUT_SCALE:-}" ]; then
-        mock_services_opts="$mock_services_opts --rpc_fanout_scale=${RPC_FANOUT_SCALE}"
-        echo "RPC fanout scale override: $RPC_FANOUT_SCALE"
-    fi
-    # Diagnostic / isolation knob. When LEAFNODE_USE_LEGACY_SLEEP=1, force
-    # LeafNodeRank to take the legacy folly::futures::sleep path even
-    # though --rpc_dist_path is supplied (rpc_dist.json is still resolved
-    # because DriverNodeRank's session mode needs it). Used by later diffs
-    # in the stack to integration-test without the mock_services side
-    # process. run-feedsim-multi.sh skips starting mock_services under the
-    # same env var.
-    if [ "${LEAFNODE_USE_LEGACY_SLEEP:-0}" = "1" ]; then
-        mock_services_opts="$mock_services_opts --use_legacy_sleep"
-        echo "RPC fanout: forced OFF via LEAFNODE_USE_LEGACY_SLEEP=1 (legacy sleep path)"
-    fi
-    # Per-MockServicesClient keepalive ping. Each channel issues a tiny
-    # getStatus() probe every N ms to defeat the cold-channel anti-pattern
-    # observed at low QPS (BGM saw 14x p95 cliff at q=5 without keepalive).
-    # Default 200 ms (t29/t30 calibration). Set MOCK_KEEPALIVE_INTERVAL_MS=0
-    # to disable; lower values reduce cold-start latency but add more
-    # background load on mock_services.
-    mock_keepalive_ms="${MOCK_KEEPALIVE_INTERVAL_MS:-200}"
-    if [ "${mock_keepalive_ms}" != "0" ]; then
-        mock_services_opts="$mock_services_opts --mock_keepalive_interval_ms=${mock_keepalive_ms}"
-        echo "MockServicesClient keepalive: ENABLED (interval=${mock_keepalive_ms} ms)"
-    fi
-    # TLS + wire compression on the outbound MockServicesClient channel.
-    # LeafNodeRank uses gengetopt (rejects unknown CLI flags), so these knobs
-    # are plumbed via env vars MOCK_TLS / MOCK_COMPRESS_ZSTD read inside
-    # MockServicesClient.cc. FEEDSIM_TLS defaults to 1 (matches prod's
-    # Rocket-over-TLS); set FEEDSIM_TLS=0 to disable. Server-side
-    # --tls_cert/--tls_key wiring is in run-feedsim-multi.sh.
-    # FEEDSIM_NO_RPC_ZSTD=1 disables ZSTD (binary default is on); leave
-    # unset for prod-parity.
-    if [ "${FEEDSIM_TLS:-1}" = "1" ]; then
+    local mock_services_opts="--rpc_dist_path=$rpc_dist_json"
+    mock_services_opts="$mock_services_opts --mock_services_host=localhost"
+    mock_services_opts="$mock_services_opts --mock_services_port=${mock_port}"
+    mock_services_opts="$mock_services_opts --rpc_fanout_scale=${rpc_fanout_scale}"
+    mock_services_opts="$mock_services_opts --mock_keepalive_interval_ms=${mock_keepalive_interval_ms}"
+    # MOCK_TLS env consumed by MockServicesClient::ctor (envBoolTrue("MOCK_TLS", false)).
+    # When the user-facing --mock-tls flag is set to 1 (default), we export
+    # both MOCK_TLS=1 (client side) and let FEEDSIM_TLS default to 1 so
+    # run-feedsim-multi.sh starts mock_services with --tls_cert/--tls_key.
+    if [ "$mock_tls" = "1" ]; then
         export MOCK_TLS=1
-        echo "MockServicesClient TLS: ENABLED (via MOCK_TLS env)"
-    fi
-    if [ "${FEEDSIM_NO_RPC_ZSTD:-0}" = "1" ]; then
-        export MOCK_COMPRESS_ZSTD=0
-        echo "MockServicesClient ZSTD: DISABLED (via MOCK_COMPRESS_ZSTD env)"
-    fi
-
-    # t43 knobs (2026-06-10): three independent knobs for the bench-vs-prod
-    # Compression / Encryption rebalance. See plan doc t41/t43 progress logs.
-    #
-    # MOCK_ZSTD_FRAC: float in [0.0, 1.0]. Fraction of MockServicesClient
-    # channels that enable per-channel ZSTD. Replaces all-or-nothing
-    # MOCK_COMPRESS_ZSTD with prod-realistic partial enablement (some
-    # downstream services compress, others don't).
-    if [ -n "${MOCK_ZSTD_FRAC:-}" ]; then
-        export MOCK_ZSTD_FRAC
-        echo "MockServicesClient ZSTD fraction: ${MOCK_ZSTD_FRAC} (overrides MOCK_COMPRESS_ZSTD)"
     fi
-    # FEEDSIM_SERVER_ZSTD: 0 disables server-side response ZSTD
-    # (compressThrift / compressPayload return passthrough). Default 1
-    # preserves current behavior. Use to reduce the bench's Compression
-    # CPU share when over-target.
-    if [ "${FEEDSIM_SERVER_ZSTD:-1}" != "1" ]; then
+    # MOCK_ZSTD_FRAC env consumed by MockServicesClient::resolveZstdFraction.
+    # Always export so the t43 c7 default 0.75 reaches the binary.
+    export MOCK_ZSTD_FRAC="$mock_zstd_frac"
+    echo "MockServicesClient: TLS=${mock_tls} ZSTD_frac=${mock_zstd_frac} keepalive_ms=${mock_keepalive_interval_ms} fanout_scale=${rpc_fanout_scale}"
+
+    # Server-side response compression. FEEDSIM_SERVER_ZSTD=0 disables
+    # compressThrift/compressPayload (server bytes emitted uncompressed).
+    # Default ON in the C++ source; we export "0" when user passes
+    # --server-zstd=0 (the t43 c7 default).
+    if [ "$server_zstd" != "1" ]; then
         export FEEDSIM_SERVER_ZSTD=0
         echo "Server-side response ZSTD: DISABLED (FEEDSIM_SERVER_ZSTD=0)"
-    fi
-    # FEEDSIM_DRIVER_TLS: 1 enables TLS on the driver↔server channel
-    # (DriverNodeRank ↔ LeafNodeRank). Server reads FEEDSIM_TLS_CERT /
-    # FEEDSIM_TLS_KEY env vars (set here to the existing bench cert/key
-    # under ${FEEDSIM_ROOT}/certs/); driver reads FEEDSIM_DRIVER_TLS
-    # directly. Closes the bench's Encryption CPU undershoot (prod
-    # 3.3-3.6% vs bench 0.9-1.5% in t41). Independent of FEEDSIM_TLS,
-    # which only covers the mock_services channel.
-    if [ "${FEEDSIM_DRIVER_TLS:-0}" = "1" ]; then
-        export FEEDSIM_DRIVER_TLS=1
-        export FEEDSIM_TLS_CERT="${FEEDSIM_ROOT}/certs/example.crt"
-        export FEEDSIM_TLS_KEY="${FEEDSIM_ROOT}/certs/example.key"
-        if [ ! -r "${FEEDSIM_TLS_CERT}" ] || [ ! -r "${FEEDSIM_TLS_KEY}" ]; then
-            echo "ERROR: FEEDSIM_DRIVER_TLS=1 but ${FEEDSIM_TLS_CERT} or .key not readable" >&2
-            exit 1
-        fi
-        echo "Driver↔Server TLS: ENABLED (cert=${FEEDSIM_TLS_CERT})"
+    else
+        echo "Server-side response ZSTD: ENABLED"
     fi
 
     # OMP_NUM_THREADS=1: cap PyTorch's OpenMP parallel backend pool to
@@ -931,8 +953,7 @@ main() {
 
     # SLA target for search_qps (95p latency in milliseconds). Default 700ms
     # matches the prod multifeed aggregator's own end-to-end budget at p95.
-    # Override via FEEDSIM_SLA_P95_MS env var.
-    sla_p95_ms="${FEEDSIM_SLA_P95_MS:-700}"
+    # Override via --sla-p95-ms CLI flag (handled in arg parsing above).
     sla_arg="95p:${sla_p95_ms}"
 
     if [ -z "$fixed_qps" ] && [ "$auto_driver_threads" != "1" ]; then