Skip to content

Commit 70d281a

Browse files
excelle08meta-codesync[bot]
authored andcommitted
Wire breakdown.csv into feedsim_autoscale_dlrm + add preprocessing/postprocessing logs (#736)
Summary: Pull Request resolved: #736 Three related changes so perfpub can identify the actual benchmarking phase and report metrics filtered to that window (instead of averaging across warmup + drain + idle periods). 1. jobs.yml: add `benchmarks/feedsim/breakdown.csv` to the copymove `after:` list for feedsim_autoscale_dlrm. The other feedsim jobs (search-mode, mini, etc.) already had it; only the DLRM autoscale variant was missing. With this, perfpub's automatic breakdown-based metric filtering kicks in for `./perfpub --no-xdb --no-manifold --dir benchmark_metrics_<id>` (see perfpub/README.md "Automatic Metric Filtering with breakdown.csv"). 2. run.sh: add preprocessing/postprocessing log entries around the existing search_qps.sh main_benchmark window. search_qps.sh already logs main_benchmark start/end around `sleep $experiment_time` (the actual measurement window); this commit adds the surrounding phase info so the full timeline is visible in breakdown.csv: - preprocessing = run.sh start → driver launch (server bring-up, graph/model load, warmup) - main_benchmark = experiment_time window (unchanged) - postprocessing = driver exit → run.sh exit (queue drain, leaf shutdown) perfpub uses only main_benchmark for default filtering; pre/post are informational for humans inspecting the CSV. 3. runtime_breakdown_utils.sh: make `create_breakdown_csv` idempotent (skip if file exists). Without this, multi-instance feedsim runs (run-feedsim-multi.sh launching N instances of run.sh) race on truncating the CSV — the second instance's create_breakdown_csv call would silently drop the first instance's already-logged entries. Verified t20 q=100/q=120 runs now record both inst1+inst2 preprocessing/main_benchmark/postprocessing entries cleanly. benchpress's copymove hook (is_move: true) moves the file out between iterations, so stale data across runs is not a concern. Verified on rtptest3440 BGM (qps=100, 120 each at 180s/300s): - breakdown.csv lands in benchmark_metrics_<id>/ after each run - `./perfpub --no-xdb --no-manifold --dir benchmark_metrics_<id>` succeeds and emits overall-metrics.csv with the breakdown-filtered window - All 4 expected operation entries present per instance: preprocessing start/end, main_benchmark start/end, postprocessing start/end (no race-induced drops) Reviewed By: YifanYuan3 Differential Revision: D105756766 fbshipit-source-id: 7ce888f9d8e962a0c255c16992ead4ece23879a5
1 parent e013cca commit 70d281a

3 files changed

Lines changed: 22 additions & 0 deletions

File tree

benchpress/config/jobs.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -781,6 +781,7 @@
781781
- 'benchmarks/feedsim/feedsim_results*.txt'
782782
- 'benchmarks/feedsim/feedsim-multi-inst-*.log'
783783
- 'benchmarks/feedsim/src/perf.data'
784+
- 'benchmarks/feedsim/breakdown.csv'
784785
- '/tmp/feedsim_log.txt'
785786

786787
- name: feedsim_autoscale_dlrm_mini

packages/common/runtime_breakdown_utils.sh

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,16 @@ create_breakdown_csv() {
2525

2626
local csv_file="${folder_path}/${breakdown_file_name}"
2727

28+
# Idempotent: if the file already exists, leave it alone. Multi-instance
29+
# workloads (e.g. feedsim run-feedsim-multi.sh) have several processes
30+
# racing to call create_breakdown_csv concurrently — truncating after the
31+
# first writer would silently drop entries already logged by the earlier
32+
# instance. benchpress's copymove hook (is_move: true) clears the file
33+
# between iterations, so stale data is not a risk.
34+
if [ -f "$csv_file" ]; then
35+
return 0
36+
fi
37+
2838
# Create CSV file with headers
2939
if echo "operation_name,PID,timestamp_type,timestamp,sub_operation_name" > "$csv_file"; then
3040
echo "Created breakdown CSV file: $csv_file"

packages/feedsim/run.sh

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -574,6 +574,12 @@ main() {
574574
fi
575575

576576
create_breakdown_csv "$BREAKDOWN_FOLDER"
577+
# Phase timeline (per-instance, shared CSV in $BREAKDOWN_FOLDER):
578+
# preprocessing = run.sh start → driver launch (server bring-up, graph/model load, warmup)
579+
# main_benchmark = the experiment_time window (logged by search_qps.sh around `sleep $experiment_time`)
580+
# postprocessing = driver exit → run.sh exit (queue drain, leaf shutdown)
581+
# perfpub reads only main_benchmark entries by default; pre/post are informational.
582+
log_preprocessing_start "$BREAKDOWN_FOLDER" "$$"
577583

578584
set -u # Enable unbound variables check from here onwards
579585

@@ -822,6 +828,9 @@ main() {
822828
no_retry_args="-N"
823829
fi
824830

831+
# Preprocessing complete; search_qps.sh will own the main_benchmark phase.
832+
log_preprocessing_end "$BREAKDOWN_FOLDER" "$$"
833+
825834
if [ -z "$fixed_qps" ] && [ "$auto_driver_threads" != "1" ]; then
826835
benchreps_tell_state "before search_qps"
827836
# shellcheck disable=SC2086
@@ -876,8 +885,10 @@ main() {
876885
benchreps_tell_state "after fixed_qps_exp"
877886
fi
878887

888+
log_postprocessing_start "$BREAKDOWN_FOLDER" "$$"
879889
sleep "$queue_drain_time" # wait for queue to drain
880890
kill -SIGINT $LEAF_PID || true > /dev/null # SIGINT so exits cleanly
891+
log_postprocessing_end "$BREAKDOWN_FOLDER" "$$"
881892
}
882893

883894
main "$@"

0 commit comments

Comments
 (0)