Skip to content

Commit 07fd003

Browse files
excelle08meta-codesync[bot]
authored andcommitted
Migrate compression to ManagedCompression (#722)
Summary: Pull Request resolved: #722 Migrate the three ZSTD compression callsites in `LeafNodeRank.cc` from raw `folly::compression::getCodec(CodecType::ZSTD)` to ManagedCompression, the documented Meta standard for application-level compression in fbcode (per `fbcode/.llms/rules/managed_compression.md` and the `managed_compression_integration` skill). ManagedCompression handles dictionary training, parameter tuning, and rollout via infrastructure rather than hard-coded codec choice/level. Three callsites migrated, two categories: - `compressPayload` (line ~399) — `leaf_random_string` category - `decompressPayload` (line ~409) — `leaf_random_string` category (same as compressPayload, since both operate on the same pseudo-random payload bytes — required so ManagedCompression serves the right dictionary on decompress) - `compressThrift` (line ~416) — `leaf_thrift_payload` category (serialized RankingResponse / CompactProtocol — distinct payload shape) Following the canonical pattern from `common/managed_compression/examples/ManagedCompressionExample.cpp`: - One process-wide `folly::Singleton<ManagedCompressionFactory>` keyed by oncall=`chips_dcperf` and project=`feedsim`. Constructing a factory per call is explicitly discouraged. - `getCachedCodec(category)` per category, preferred over `getCodec()` for hot paths. - 2 categories, both clearly distinct payload shapes (random bytes vs thrift CompactProtocol). Per the skill, category count is kept modest. Open-source build path: the benchpress repo is open-sourced and ManagedCompression is internal-only, so the include and use sites are gated behind `#ifdef BENCHPRESS_INTERNAL`. When the gate is undefined (the current OSS / CMake build path that all install scripts use today), the original raw folly ZSTD code remains as the fallback so the OSS build still works. `LeafNodeRank.cc` has no Buck build target — it is built only by CMake at fbpkg-install time — so no Buck-side wiring is needed in this commit. A follow-up (Phase 6) can add `-DBENCHPRESS_INTERNAL=1` to the internal CMake invocation to flip the gate on. Stack position: depends on `bebd655f6d` (Phase 4-B thrift structs). Sibling of Phase 5-A mock_services (`bd2517aa83`). Reviewed By: charles-typ Differential Revision: D103768051 fbshipit-source-id: 307a9a1e9d939a7feeb0a673997824134eaf792c
1 parent 62a2b7f commit 07fd003

1 file changed

Lines changed: 71 additions & 1 deletion

File tree

packages/feedsim/third_party/src/workloads/ranking/LeafNodeRank.cc

Lines changed: 71 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,21 @@
2828
#include <folly/Range.h>
2929
#include <folly/compression/Compression.h>
3030
#include <folly/executors/CPUThreadPoolExecutor.h>
31+
32+
// ManagedCompression is the documented Meta standard for application-level
33+
// compression (per fbcode/.llms/rules/managed_compression.md), but it is
34+
// internal-only. The benchpress repo is open-sourced, so the include and
35+
// usages below are gated behind BENCHPRESS_INTERNAL. When the gate is not
36+
// defined (the current OSS / CMake build path) we fall back to raw folly
37+
// ZSTD, which is the historical behavior. The gate is wired up by the
38+
// fbcode-internal Buck build (see packages/feedsim/.../mock_services/BUCK
39+
// for the analogous internal-only wiring); the open-source CMake build
40+
// leaves it undefined.
41+
#ifdef BENCHPRESS_INTERNAL
42+
#include <folly/Singleton.h>
43+
#include "common/managed_compression/ManagedCompression.h"
44+
#endif
45+
3146
#include <folly/executors/GlobalExecutor.h>
3247
#include <folly/executors/IOThreadPoolExecutor.h>
3348
#include <folly/futures/Future.h>
@@ -396,29 +411,84 @@ void ThreadStartup(
396411
}
397412
}
398413

399-
std::string compressPayload(const std::string& data, int result) {
414+
#ifdef BENCHPRESS_INTERNAL
415+
namespace {
416+
using facebook::managed_compression::ManagedCompressionFactory;
417+
418+
// One ManagedCompressionFactory per (oncall, project) pair, lifetime =
419+
// process. Per the ManagedCompression skill / wiki, constructing a new
420+
// factory per call is expensive and explicitly discouraged.
421+
//
422+
// Two categories are used in this file:
423+
// "leaf_random_string" — the pseudo-random payload bytes shared by
424+
// compressPayload / decompressPayload. Both
425+
// sides MUST use the same category so
426+
// ManagedCompression can serve the right
427+
// dictionary on decompress.
428+
// "leaf_thrift_payload" — serialized RankingResponse (CompactProtocol)
429+
// consumed by compressThrift.
430+
class FeedSimCompressionTag {};
431+
folly::Singleton<ManagedCompressionFactory, FeedSimCompressionTag> gFactory(
432+
[] {
433+
return new ManagedCompressionFactory(
434+
/*oncall_team=*/"chips_dcperf",
435+
/*project=*/"feedsim");
436+
});
437+
438+
std::shared_ptr<folly::compression::Codec> getRandomStringCodec() {
439+
// getCachedCodec() reuses the codec instance per category; preferred
440+
// over getCodec() for hot paths per references/cpp.md. try_get() can
441+
// return null before SingletonVault::registrationComplete() or after
442+
// destroyInstances() during shutdown — fail loudly rather than crash
443+
// with a null-deref.
444+
auto factory = gFactory.try_get();
445+
CHECK(factory) << "ManagedCompressionFactory singleton unavailable";
446+
return factory->getCachedCodec("leaf_random_string");
447+
}
448+
449+
std::shared_ptr<folly::compression::Codec> getThriftPayloadCodec() {
450+
auto factory = gFactory.try_get();
451+
CHECK(factory) << "ManagedCompressionFactory singleton unavailable";
452+
return factory->getCachedCodec("leaf_thrift_payload");
453+
}
454+
} // namespace
455+
#endif // BENCHPRESS_INTERNAL
456+
457+
std::string compressPayload(const std::string& data, int /*result*/) {
400458
folly::StringPiece output(
401459
data.data(),
402460
std::min(args.compression_data_size_arg, args.random_data_size_arg));
461+
#ifdef BENCHPRESS_INTERNAL
462+
return getRandomStringCodec()->compress(output);
463+
#else
403464
auto codec =
404465
folly::compression::getCodec(folly::compression::CodecType::ZSTD);
405466
std::string compressed = codec->compress(output);
406467
return std::move(compressed);
468+
#endif
407469
}
408470

409471
std::string decompressPayload(const std::string& data) {
472+
#ifdef BENCHPRESS_INTERNAL
473+
return getRandomStringCodec()->uncompress(data);
474+
#else
410475
auto codec =
411476
folly::compression::getCodec(folly::compression::CodecType::ZSTD);
412477
std::string decompressed = codec->uncompress(data);
413478
return decompressed;
479+
#endif
414480
}
415481

416482
std::unique_ptr<folly::IOBuf> compressThrift(
417483
std::unique_ptr<folly::IOBuf> buf) {
484+
#ifdef BENCHPRESS_INTERNAL
485+
return getThriftPayloadCodec()->compress(buf.get());
486+
#else
418487
auto codec =
419488
folly::compression::getCodec(folly::compression::CodecType::ZSTD);
420489
auto compressed_buf = codec->compress(buf.get());
421490
return compressed_buf;
491+
#endif
422492
}
423493

424494
folly::IOBufQueue serializePayload(const ranking::RankingResponse& resp) {

0 commit comments

Comments
 (0)