Skip to content

bug: --full flag silently ignored during benchmark — per-transaction .csv.zst never written #476

@russfellows

Description

@russfellows

Bug: --full flag silently ignored — per-transaction .csv.zst file never written

Description

Running any benchmark with the --full flag does not produce a per-transaction
.csv.zst output file. Only the .json.zst aggregate is written, making post-hoc
per-operation analysis impossible. The flag appears to be accepted without error but
has no effect.

Steps to Reproduce

warp put --host=<host> --access-key=<key> --secret-key=<secret> --full

Expected: Two output files written after the benchmark:

  • warp-put-<timestamp>.csv.zst — one row per operation (full per-transaction log)
  • warp-put-<timestamp>.json.zst — aggregated summary

Actual: Only warp-put-<timestamp>.json.zst is written. The .csv.zst file is
never created.

Root Cause

Introduced in commit 483f844, which added addCollector(). That function was
written to gate per-transaction collection on ctx.Bool("full"), but the --full
flag was only registered in analyzeFlags — never added to benchFlags. As a
result, ctx.Bool("full") always returns false during benchmark execution, and
the collector always falls back to EmptyOpsCollector (no ops stored, no file
written).

// addCollector() — intended gate:
if ctx.Bool("full") {        // always false during benchmark — flag not registered
    // fan ops to NewOpsCollector + LiveCollector
} else {
    return bench.EmptyOpsCollector, updates  // always taken
}

Impact

  • warp analyze --full <file> (full per-operation latency/throughput analysis) is
    effectively unreachable — there is no .csv.zst file to analyze.
  • Users explicitly requesting --full for detailed post-benchmark analysis get
    silently degraded output with no warning or error.
  • warp merge, warp cmp, and per-request filtering (--analyze.op,
    --analyze.host, --analyze.skip) all require a .csv.zst — none of these
    workflows are usable.

Additional Finding

Running warp analyze file.csv.zst without --full re-aggregates individual
operations into 1-second buckets before reporting — producing results equivalent to
reading the .json.zst aggregate — with no indication to the user that per-operation
precision has been lost.

Fix

A fix is available in PR #475, which:

  1. Makes addCollector() correctly gate on --full (flag already present in
    analyzeFlags, which is combined with benchFlags for all benchmark commands).
  2. Makes --full additive: both .csv.zst and .json.zst are always written
    together when the flag is set.
  3. Mirrors the fix in benchserver.go for distributed mode.
  4. Adds a warning in warp analyze when a .csv.zst is analyzed without --full,
    directing users to the more accurate analysis path.
  5. Updates README documentation and adds 7 unit tests.

PR: #475

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions