Match prod multifeed RPC profile: ZSTD, smaller response, DLRM=1, feature extractors (#744)

Summary:
Pull Request resolved: #744

Coordinated changes to close the apples-to-apples gap between FeedSim's t30 hot-function breakdown and prod `multifeed/aggregator_main`. Source comparison anchored on the t30 last-known-good baseline recorded in T269255604.

1. **`dlrm_inferences` 2 → 1** in `feedsim_dlrm` and `feedsim_autoscale_dlrm`. Profile shows FeedSim was +9.9pp Ranking-Prediction over prod (FS 15.8% vs prod 5.9% on BGM at dlrm=2). Halving inferences-per-request brings Ranking-Prediction toward the ~8% mark.

2. **Feature extractors enabled with t29-mini sizing** in `feedsim_autoscale_dlrm`. Changed args from `'--feature-extractors={feature_extractors}'` (with var `feature_extractors=0` → flag disabled) to boolean `'--feature-extractors'` and bumped `num_stories` 100 → 1800 to match the t29 mini tuning (90K extractor calls/req that landed FE CPU share in prod's 30-35% band).

3. **ZSTD wire compression on RocketClientChannel.** `channel->setDesiredCompressionConfig(zstdConfig)` on the outbound mock_services client. Plumbed via `MOCK_COMPRESS_ZSTD` env var (default on; set =0 to disable). LeafNodeRank uses gengetopt for CLI which rejects unknown flags, so env-var is the only injection path. Closes the −2.8pp RPC-Compression gap.

4. **Smaller response payload** in `RankingGenerators.h`. `RankingObject::write` self-time was 54% of FeedSim's RPC-Serialization bucket (+12.6pp over prod). Hardcoded sizes moved to named constexprs: `kPayloadEntriesPerMap` 5 → 2, `kActionsPerObject` 5 → 2, `kObjectsPerStory` 20 → 10. Net ~70% reduction in per-response serialization work.

5. **`FEEDSIM_SLA_P95_MS` env var** added to `run.sh` so search_qps SLA can be overridden without on-host sed-patching. Defaults to 500ms; t29/t30 used 700ms to give the new high-fanout design more headroom than the legacy 200ms-fixed-sleep code path.

6. **`-luring` baked into install scripts** (`FS_LDFLAGS`) to work around folly's CMake not propagating liburing as a transitive link dependency on RHEL9 + liburing 2.12 (undef refs: `io_uring_register_ifq`, `io_uring_register_eventfd`).

7. **Example TLS certs extracted from `packages/common/certs.tar.gz`** during install (--strip-components=1). Cert plumbing left in place for a future TLS-on-Rocket follow-up.

**TLS NOT enabled** in this diff: the open-source FBThrift v2026.01.05.00 Rocket transport rejects plain `folly::AsyncSSLSocket` — needs a fizz-based AsyncTransport adapter that isn't trivially exposed. The probe RPC inside MockServicesClient times out after 30s, LeafNodeRank catches and falls back to legacy folly::futures::sleep mode (no fanout, no TLS), which produces a misleading profile. Closing the Encryption gap (prod 3.3% vs FS 0%) is split into a follow-up that wires fizz properly into Rocket.

Reviewed By: YifanYuan3

Differential Revision: D107318437

Author: excelle08

benchpress/config/jobs.yml

@@ -659,7 +659,7 @@
     - 'dlrm_model=models/dlrm_small.pt'
     - 'dlrm_batch_size=256'
     - 'dlrm_threads=1'
-    - 'dlrm_inferences=64'
+    - 'dlrm_inferences=1'
     - 'client_side_features=0'
     - 'client_batch_size=256'
     - 'client_inferences=64'
@@ -739,7 +739,7 @@
     - '--client-feature-seed={client_feature_seed}'
     - '--client-num-dense={client_num_dense}'
     - '--client-num-sparse={client_num_sparse}'
-    - '--feature-extractors={feature_extractors}'
+    - '--feature-extractors'
     - '--feature-complexity={feature_complexity}'
     - '--num-stories={num_stories}'
     - '--extractors-per-story={extractors_per_story}'
@@ -752,19 +752,26 @@
     - 'io_mean=200'
     - 'workload=dlrm'
     - 'dlrm_model=models/dlrm_small.pt'
-    - 'dlrm_batch_size=256'
+    # 64 matches t29 baseline (was 256, which is 4x oversized vs t29). Combined
+    # with dlrm_inferences=1 the per-request DLRM CPU drops further toward
+    # prod multifeed's Ranking-Prediction share (~6%).
+    - 'dlrm_batch_size=64'
     - 'dlrm_threads=1'
-    - 'dlrm_inferences=64'
+    - 'dlrm_inferences=1'
     - 'client_side_features=0'
     - 'client_batch_size=256'
     - 'client_inferences=64'
     - 'client_feature_seed=42'
     - 'client_num_dense=13'
     - 'client_num_sparse=26'
-    - 'feature_extractors=0'
     - 'feature_complexity=5'
-    - 'num_stories=100'
-    - 'extractors_per_story=50'
+    # 400 x 400 = 160K extractor calls/req. Matches the t29 BGM/CPL/GRC
+    # search_qps validation that converged cleanly at 700ms p95 SLA. The
+    # earlier 1800x50 tuning came from feedsim_autoscale_dlrm_mini (fixed
+    # QPS, no search) and produces queueing collapse under search_qps.
+    # 5K calls/req (100x50) is too few to surface FeatureExtraction CPU.
+    - 'num_stories=400'
+    - 'extractors_per_story=400'
     - 'silesia_dir=silesia'
     - 'stories_per_request=10'
     - 'extra_args='

packages/feedsim/install_feedsim.sh

@@ -45,7 +45,7 @@ dnf install -y bc ninja-build flex bison git texinfo binutils-devel \
     libsodium-devel libunwind-devel bzip2-devel double-conversion-devel \
     libzstd-devel lz4-devel xz-devel snappy-devel libtool bzip2 openssl-devel \
     zlib-devel libdwarf libdwarf-devel libaio-devel libatomic patch jq \
-    xxhash xxhash-devel unzip
+    xxhash xxhash-devel unzip liburing-devel
 
 # Creates feedsim directory under benchmarks/
 mkdir -p "${BENCHPRESS_ROOT}/benchmarks/feedsim"
@@ -246,6 +246,21 @@ else
     msg "[SKIPPED] Silesia corpus already present at $SILESIA_DIR"
 fi
 
+# Extract example TLS certs for mock_services (used when --tls_cert/--tls_key
+# are passed; see run-feedsim-multi.sh). The tarball ships example.crt and
+# example.key suitable for benchmark use only (no peer verification).
+CERTS_DIR="${FEEDSIM_ROOT_SRC}/certs"
+CERTS_TARBALL="${BENCHPRESS_ROOT}/packages/common/certs.tar.gz"
+if [ -f "$CERTS_TARBALL" ]; then
+    mkdir -p "$CERTS_DIR"
+    # --strip-components=1 drops the top-level `certs/` directory inside the
+    # tarball so the files land directly at $CERTS_DIR/example.{crt,key}.
+    tar -xzf "$CERTS_TARBALL" -C "$CERTS_DIR" --strip-components=1
+    msg "Extracted TLS certs to $CERTS_DIR"
+else
+    msg "[WARNING] $CERTS_TARBALL not found; TLS for mock_services will be unavailable"
+fi
+
 # Installing FeedSim
 cd "${FEEDSIM_ROOT_SRC}"
 
@@ -289,7 +304,12 @@ mkdir -p build && cd build/
 # Build FeedSim with DLRM support
 FS_CFLAGS="${BP_CFLAGS:--O3 -DNDEBUG}"
 FS_CXXFLAGS="${BP_CXXFLAGS:--O3 -DNDEBUG }"
-FS_LDFLAGS="${BP_LDFLAGS:-} -latomic -Wl,--export-dynamic"
+# -luring: folly's IoUringZeroCopyBufferPool/IoUringEvent reference io_uring
+# symbols but folly's CMake doesn't propagate liburing as a transitive link
+# dependency. On RHEL9 + liburing 2.12 the static libfolly.a otherwise fails
+# to link LeafNodeRank with undefined references to io_uring_register_ifq /
+# io_uring_register_eventfd. Force the link explicitly.
+FS_LDFLAGS="${BP_LDFLAGS:-} -luring -latomic -Wl,--export-dynamic"
 
 cmake -G Ninja \
     -DCMAKE_BUILD_TYPE=Release \

packages/feedsim/install_feedsim_aarch64.sh

@@ -51,7 +51,7 @@ dnf install -y bc ninja-build flex bison git texinfo binutils-devel \
     libsodium-devel libunwind-devel bzip2-devel double-conversion-devel \
     libzstd-devel lz4-devel xz-devel snappy-devel libtool bzip2 openssl-devel \
     zlib-devel libdwarf libdwarf-devel libaio-devel libatomic patch jq \
-    xxhash xxhash-devel unzip
+    xxhash xxhash-devel unzip liburing-devel
 
 # Creates feedsim directory under benchmarks/
 mkdir -p "${BENCHPRESS_ROOT}/benchmarks/feedsim"
@@ -304,6 +304,21 @@ else
     msg "[SKIPPED] Silesia corpus already present at $SILESIA_DIR"
 fi
 
+# Extract example TLS certs for mock_services (used when --tls_cert/--tls_key
+# are passed; see run-feedsim-multi.sh). The tarball ships example.crt and
+# example.key suitable for benchmark use only (no peer verification).
+CERTS_DIR="${FEEDSIM_ROOT_SRC}/certs"
+CERTS_TARBALL="${BENCHPRESS_ROOT}/packages/common/certs.tar.gz"
+if [ -f "$CERTS_TARBALL" ]; then
+    mkdir -p "$CERTS_DIR"
+    # --strip-components=1 drops the top-level `certs/` directory inside the
+    # tarball so the files land directly at $CERTS_DIR/example.{crt,key}.
+    tar -xzf "$CERTS_TARBALL" -C "$CERTS_DIR" --strip-components=1
+    msg "Extracted TLS certs to $CERTS_DIR"
+else
+    msg "[WARNING] $CERTS_TARBALL not found; TLS for mock_services will be unavailable"
+fi
+
 # Installing FeedSim
 cd "${FEEDSIM_ROOT_SRC}/src"
 
@@ -349,7 +364,7 @@ mkdir -p build && cd build/
 # Build FeedSim
 FS_CFLAGS="${BP_CFLAGS:--O3 -DNDEBUG}"
 FS_CXXFLAGS="${BP_CXXFLAGS:--O3 -DNDEBUG -Wno-deprecated-declarations}"
-FS_LDFLAGS="${BP_LDFLAGS:-} -latomic -Wl,--export-dynamic -L${FEEDSIM_THIRD_PARTY_SRC}/miniconda3/lib -Wl,-rpath,${FEEDSIM_THIRD_PARTY_SRC}/miniconda3/lib"
+FS_LDFLAGS="${BP_LDFLAGS:-} -luring -latomic -Wl,--export-dynamic -L${FEEDSIM_THIRD_PARTY_SRC}/miniconda3/lib -Wl,-rpath,${FEEDSIM_THIRD_PARTY_SRC}/miniconda3/lib"
 
 BP_CC="${BP_CC:-gcc}"
 BP_CXX="${BP_CXX:-g++}"

packages/feedsim/install_feedsim_aarch64_ubuntu.sh

@@ -36,6 +36,28 @@ apt install -y bc cmake ninja-build flex bison texinfo binutils-dev \
     zlib1g-dev libdwarf-dev libaio-dev libatomic1 patch perl libiberty-dev \
     sysstat jq unzip xxhash libxxhash-dev libboost-all-dev
 
+# Install liburing >= 2.6 from source. Ubuntu's apt-shipped liburing is
+# older than folly's minimum, so folly's io_uring integration links
+# `-luring` against a nonexistent library. Build tag liburing-2.12 upstream
+# and install into /usr/local so folly's `find_library` picks it up.
+if ! ldconfig -p 2>/dev/null | grep -q "liburing.so.2"; then
+    msg "Building liburing 2.12 from source..."
+    LIBURING_BUILD_DIR="${FEEDSIM_THIRD_PARTY_SRC}/liburing_build"
+    mkdir -p "${LIBURING_BUILD_DIR}"
+    if ! [ -d "${LIBURING_BUILD_DIR}/liburing" ]; then
+        git clone --depth 1 --branch liburing-2.12 \
+            https://github.com/axboe/liburing.git \
+            "${LIBURING_BUILD_DIR}/liburing"
+    fi
+    (
+        cd "${LIBURING_BUILD_DIR}/liburing"
+        ./configure --prefix=/usr/local
+        make -j"$(nproc)"
+        make install
+    )
+    ldconfig
+fi
+
 # Creates feedsim directory under benchmarks/
 mkdir -p "${BENCHPRESS_ROOT}/benchmarks/feedsim"
 cd "${BENCHPRESS_ROOT}/benchmarks"

packages/feedsim/install_feedsim_ubuntu.sh

@@ -32,6 +32,28 @@ apt install -y bc cmake ninja-build flex bison texinfo binutils-dev \
     zlib1g-dev libdwarf-dev libaio-dev libatomic1 patch perl libiberty-dev \
     sysstat jq xxhash libxxhash-dev unzip
 
+# Install liburing >= 2.6 from source. Ubuntu's apt-shipped liburing (0.7 on
+# 20.04, 2.1 on 22.04) is older than folly's minimum, so folly's io_uring
+# integration links `-luring` against a nonexistent library. Build tag
+# liburing-2.12 upstream and install into /usr/local so folly's `find_library`
+# picks it up ahead of the system one.
+if ! ldconfig -p 2>/dev/null | grep -q "liburing.so.2"; then
+    msg "Building liburing 2.12 from source..."
+    LIBURING_BUILD_DIR="${FEEDSIM_THIRD_PARTY_SRC}/liburing_build"
+    mkdir -p "${LIBURING_BUILD_DIR}"
+    if ! [ -d "${LIBURING_BUILD_DIR}/liburing" ]; then
+        git clone --depth 1 --branch liburing-2.12 \
+            https://github.com/axboe/liburing.git \
+            "${LIBURING_BUILD_DIR}/liburing"
+    fi
+    (
+        cd "${LIBURING_BUILD_DIR}/liburing"
+        ./configure --prefix=/usr/local
+        make -j"$(nproc)"
+        make install
+    )
+    ldconfig
+fi
 
 # Creates feedsim directory under benchmarks/
 mkdir -p "${BENCHPRESS_ROOT}/benchmarks/feedsim"

packages/feedsim/run.sh

@@ -729,6 +729,16 @@ main() {
         mock_services_opts="$mock_services_opts --mock_keepalive_interval_ms=${MOCK_KEEPALIVE_INTERVAL_MS}"
         echo "MockServicesClient keepalive: ENABLED (interval=${MOCK_KEEPALIVE_INTERVAL_MS} ms)"
     fi
+    # Wire compression on the outbound MockServicesClient channel.
+    # LeafNodeRank uses gengetopt (rejects unknown flags), so the knob is
+    # plumbed via the MOCK_COMPRESS_ZSTD env var read inside
+    # MockServicesClient.cc — not as a CLI flag. FEEDSIM_NO_RPC_ZSTD=1 turns
+    # ZSTD off (binary default is on); leave unset for default-on parity with
+    # prod's wire-compressed channels.
+    if [ "${FEEDSIM_NO_RPC_ZSTD:-0}" = "1" ]; then
+        export MOCK_COMPRESS_ZSTD=0
+        echo "MockServicesClient ZSTD: DISABLED (via MOCK_COMPRESS_ZSTD env)"
+    fi
 
     # OMP_NUM_THREADS=1: cap PyTorch's OpenMP parallel backend pool to
     # 1 thread. at::set_num_threads(1) only affects libtorch's native
@@ -841,10 +851,18 @@ main() {
     # Preprocessing complete; search_qps.sh will own the main_benchmark phase.
     log_preprocessing_end "$BREAKDOWN_FOLDER" "$$"
 
+    # SLA target for search_qps (95p latency in milliseconds). Default 500ms.
+    # Override via FEEDSIM_SLA_P95_MS env var. Prior experiments (t29, t30)
+    # ran at 700ms to give the system more headroom past the prod-aggregator's
+    # own end-to-end budget.
+    sla_p95_ms="${FEEDSIM_SLA_P95_MS:-500}"
+    sla_arg="95p:${sla_p95_ms}"
+
     if [ -z "$fixed_qps" ] && [ "$auto_driver_threads" != "1" ]; then
         benchreps_tell_state "before search_qps"
+        echo "search_qps SLA: ${sla_arg}"
         # shellcheck disable=SC2086
-        scripts/search_qps.sh -w 15 -f 300 -s 95p:500 -P "$LEAF_PID" -B "$BREAKDOWN_FOLDER" $qps_threshold_args $no_retry_args -o "${FEEDSIM_ROOT}/${result_filename}" -- \
+        scripts/search_qps.sh -w 15 -f 300 -s "$sla_arg" -P "$LEAF_PID" -B "$BREAKDOWN_FOLDER" $qps_threshold_args $no_retry_args -o "${FEEDSIM_ROOT}/${result_filename}" -- \
             build/workloads/ranking/DriverNodeRank \
                 --server "0.0.0.0:$port" \
                 --monitor_port "$client_monitor_port" \
@@ -856,8 +874,9 @@ main() {
         benchreps_tell_state "after search_qps"
     elif [ -z "$fixed_qps" ] && [ "$auto_driver_threads" = "1" ]; then
         benchreps_tell_state "before search_qps"
+        echo "search_qps SLA: ${sla_arg}"
         # shellcheck disable=SC2086
-        scripts/search_qps.sh -a -w 15 -f 300 -s 95p:500 -P "$LEAF_PID" -B "$BREAKDOWN_FOLDER" $qps_threshold_args $no_retry_args -o "${FEEDSIM_ROOT}/${result_filename}" -- \
+        scripts/search_qps.sh -a -w 15 -f 300 -s "$sla_arg" -P "$LEAF_PID" -B "$BREAKDOWN_FOLDER" $qps_threshold_args $no_retry_args -o "${FEEDSIM_ROOT}/${result_filename}" -- \
             build/workloads/ranking/DriverNodeRank \
                 --monitor_port "$client_monitor_port" \
                 --server "0.0.0.0:$port" \

packages/feedsim/third_party/src/workloads/ranking/MockServicesClient.cc

@@ -20,13 +20,31 @@
 #include <string>
 #include <utility>
 
+#include <cstdlib>
+#include <cstring>
+
 #include <folly/SocketAddress.h>
 #include <folly/io/async/AsyncSocket.h>
 
 #include <thrift/lib/cpp2/async/RocketClientChannel.h>
+#include <thrift/lib/thrift/gen-cpp2/RpcMetadata_types.h>
 
 #include "mock_services/gen-cpp2/MockServiceAsyncClient.h"
 
+namespace {
+// LeafNodeRank uses gengetopt (not gflags) for CLI parsing, so we cannot add
+// CLI flags here. Read knobs from env vars instead. Set MOCK_COMPRESS_ZSTD=0
+// to disable per-channel ZSTD negotiation (default on).
+bool envBoolTrue(const char* name, bool default_value) {
+  const char* v = std::getenv(name);
+  if (v == nullptr) {
+    return default_value;
+  }
+  return std::strcmp(v, "1") == 0 || std::strcmp(v, "true") == 0 ||
+      std::strcmp(v, "TRUE") == 0;
+}
+} // namespace
+
 namespace ranking {
 
 // KeepaliveTimer fires a fire-and-forget getStatus() RPC every
@@ -100,12 +118,18 @@ MockServicesClient::MockServicesClient(
   // RocketClientChannel must be created on the EventBase thread. Use
   // runInEventBaseThreadAndWait so this constructor remains usable from
   // any thread (typically the main thread during ThreadStartup).
-  evb_->runInEventBaseThreadAndWait([this, &host, port]() {
+  const bool use_zstd = envBoolTrue("MOCK_COMPRESS_ZSTD", true);
+  evb_->runInEventBaseThreadAndWait([this, &host, port, use_zstd]() {
     folly::SocketAddress addr(host, port, /*allowNameLookup=*/true);
     folly::AsyncSocket::UniquePtr socket(
         new folly::AsyncSocket(evb_, addr));
     auto channel =
         apache::thrift::RocketClientChannel::newChannel(std::move(socket));
+    if (use_zstd) {
+      apache::thrift::CompressionConfig compressionConfig;
+      compressionConfig.codecConfig().ensure().set_zstdConfig();
+      channel->setDesiredCompressionConfig(compressionConfig);
+    }
     client_ =
         std::make_unique<mock_services::MockServiceAsyncClient>(
             std::move(channel));

packages/feedsim/third_party/src/workloads/ranking/generators/RankingGenerators.h

@@ -99,6 +99,21 @@ inline ranking::RankingPayloadVecMap generateRandomVecMap(size_t length) {
   return map;
 }
 
+// Per-payload-map entry count. Cut from 5 to 2 to bring the response
+// wire-size and serialization CPU closer to prod multifeed/aggregator_main:
+// before, FS-side RPC-Serialization was 21% of CPU with
+// `RankingObject::write` self-time = 54% of that bucket; prod's analogous
+// `GetStoriesResponse` averages ~12 simple fields per story plus one binary
+// blob, not 3 typed maps × N entries. See the t30 hot-func breakdown
+// comparison (2026-06-01) for the source data.
+constexpr size_t kPayloadEntriesPerMap = 2;
+// Actions per RankingObject. Was 5; reduced to 2 (same rationale).
+constexpr size_t kActionsPerObject = 2;
+// RankingObjects per RankingStory. Was 20; reduced to 10 to roughly
+// halve the per-story serialization work without changing the request
+// fanout shape.
+constexpr size_t kObjectsPerStory = 10;
+
 inline ranking::RankingObject generateRandomRankingObject(
     size_t actions_length) {
   ranking::RankingObject obj;
@@ -109,10 +124,9 @@ inline ranking::RankingObject generateRandomRankingObject(
   obj.actorID() = static_cast<int64_t>(rand_int);
   obj.createTime() = static_cast<int64_t>(rand_int);
 
-  // FIXME(cltorres): Populate with realistic sizes
-  obj.payloadIntMap() = generateRandomIntMap(5);
-  obj.payloadStrMap() = generateRandomStringMap(5);
-  obj.payloadVecMap() = generateRandomVecMap(5);
+  obj.payloadIntMap() = generateRandomIntMap(kPayloadEntriesPerMap);
+  obj.payloadStrMap() = generateRandomStringMap(kPayloadEntriesPerMap);
+  obj.payloadVecMap() = generateRandomVecMap(kPayloadEntriesPerMap);
 
   auto actions_ref = obj.actions();
   auto& actions = *actions_ref;
@@ -132,11 +146,10 @@ inline ranking::RankingStory generateRandomRankingStory(
   auto objects_ref = story.objects();
   auto& objects = *objects_ref;
   objects.reserve(ranking_objects_length);
-  // TODO(cltorres): Determine distribution of Actions per ranking object
   std::generate_n(
       std::back_inserter(objects),
       ranking_objects_length,
-      std::bind(generateRandomRankingObject, 5));
+      std::bind(generateRandomRankingObject, kActionsPerObject));
   story.weight() = static_cast<double>(rand_int);
   story.storyType() = static_cast<ranking::RankingStoryType>(
       rand_int %
@@ -152,11 +165,10 @@ inline ranking::RankingResponse generateRandomRankingResponse(
   auto rankingStories_ref = resp.rankingStories();
   auto& rankingStories = *rankingStories_ref;
   rankingStories.reserve(ranking_stories_length);
-  // TODO(cltorres): Determine distribution of ranking objects per story
   std::generate_n(
       std::back_inserter(rankingStories),
       ranking_stories_length,
-      std::bind(generateRandomRankingStory, 20));
+      std::bind(generateRandomRankingStory, kObjectsPerStory));
   auto objectCounts_ref = resp.objectCounts();
   auto& objectCounts = *objectCounts_ref;
   objectCounts.reserve(ranking_stories_length);

packages/feedsim/third_party/src/workloads/ranking/mock_services/MockServiceMain.cc

@@ -52,6 +52,12 @@ DECLARE_int32(latency_cap_us);
 DECLARE_int32(latency_offset_us);
 DECLARE_int32(latency_skip_threshold_us);
 
+// NOTE: TLS support intentionally not wired here. The open-source FBThrift
+// v2026.01.05.00 Rocket transport requires a fizz-based AsyncTransport (not
+// plain AsyncSSLSocket) on the client side; without it, TLS-REQUIRED servers
+// reject the first Rocket setup frame before SSL handshake even begins. See
+// the Encryption gap analysis in the t31 memo for the closure plan.
+
 int main(int argc, char** argv) {
   folly::Init init(&argc, &argv);