[auto-perf-opt] Cap pk_index_parallel_compaction threads at num_cores

[luohaha]

Summary

PK index compaction is CPU-bound (in-memory SST byte-merge in KeyValueMerger::merge). When pk_index_parallel_compaction_threadpool_max_threads is configured higher than the number of cores, the extra threads add only context-switch overhead and starve co-tenant thread pools (publish_version, async_delta_writer, cloud_native_pk_index_execution) that share the same CPU.

This patch caps the effective thread count at CpuInfo::num_cores() in both calc_max_threads (init path) and update_max_threads (ADMIN SET path) so the bound cannot be bypassed at runtime. When the configured value is ≤ num_cores, behaviour is unchanged.

Bottleneck evidence (auto-perf-opt iteration 001, branch-4.1 + commit 7af3ba7, template 999_1gb_1_1tb on 6 × 32-core CN)

• All 6 BE/CN nodes pinned at 99.9-100 % non-idle CPU, load1 of 108-162 on 32 cores (3.4-5× oversubscription).
• cloud_native_pk_index_compact thread pool: cluster size 768 (= 128/CN × 6 CN, set by the benchmark template), 605 active threads, avg per-task execute time 46.6 s, queue 124. No other pool comes close.
• Every slow publish-version trace shows parallel_upsert_wait_us = 2.5-2.9 s of a 3.1 s total — the publish thread is blocked waiting on cloud_native_pk_index_execution workers, which themselves cannot be scheduled past the compaction threads.
• Upsert P99 = 3 621 ms, max = 20 092 ms versus goal of ≤ 3 s. Insert P99 ≈ upsert P99 → bottleneck is shared-CPU contention, not the PK-update inner loop.

Code change

be/src/storage/lake/lake_persistent_index_parallel_compact_mgr.cpp:

// In calc_max_threads():
max_threads = std::min(max_threads, CpuInfo::num_cores());
return std::max(1, max_threads);

// In update_max_threads(int):
max_threads = std::max(1, std::min(max_threads, CpuInfo::num_cores()));

Expected impact

With the benchmark's configured value of 128 on a 32-core CN:

• Active compaction threads: 605 cluster-wide → ≤ 192 (one per core).
• Per-thread effective CPU: ~1 core (vs ~0.32 today) → per-task time should fall, total compaction throughput should stay similar or rise (no context-switch tax).
• Freed CPU unlocks publish_version, async_delta_writer, put_aggregate_metadata, cloud_native_pk_index_execution → publish wait drops → upsert max should fall well below the 20 s observed.

Risks

If SST queue grows monotonically under the lower compaction concurrency, PK-index lookup amplification could slow the read/publish path differently. Iteration 002 of the auto-perf-opt run will measure this; if SST count grows unbounded we will revert and try a different angle (e.g. tune pk_index_early_sst_compaction_threshold).

Test plan

• Iteration 002 of the auto-perf-opt benchmark deploys this PR and re-runs the 999_1gb_1_1tb workload; expected: cluster CPU drops below 100 %, upsert max < 20 s, freshness P99 lower.
• Watch cloud_native_pk_index_compact queue_count over the test window — must not grow unbounded.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

luohaha seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[luohaha]

Superseded by #72434 — profile (perf -F 99 60s on the hottest CN) shows the actual hotspot is per-key protobuf allocator churn inside KeyValueMerger::merge/flush (38.6% self-time in my_free); throttling the compaction thread pool would have made things worse by letting SSTs accumulate. Closing in favor of the perf-driven fix.