Skip to content

benchmarking.rst

Latest commit

 

History

History
136 lines (106 loc) · 6.15 KB

benchmarking.rst

File metadata and controls

136 lines (106 loc) · 6.15 KB
Raw

Benchmarking using command-line tool

This page provides information on benchmarking BirdNET's performance on different hardware configurations. For some real world benchmark results, see :doc:`Comparative benchmarking results <benchmarking_results>`.

.. toctree::
  :hidden:

  benchmarking_results

Note

The information provided here is not up-to-date with the latest developments in the BirdNET library.

Example usage

  • Show benchmark options: birdnet-benchmark --help

  • Predict top 5 species for each segment using CPU und TFLite backend (single file): birdnet-benchmark soundscape.wav

  • Predict all audio files in a directory: birdnet-benchmark path/to/audio/files/

  • Use Protobuf backend: birdnet-benchmark soundscape.wav -b "pb"

  • Output predictions for top 10 species: birdnet-benchmark soundscape.wav --top-k 10 --confidence -100

  • Run on GPU: birdnet-benchmark soundscape.wav --backend "pb" --worker 1 --device "GPU" --batch-size 1000

    To determine the largest possible batch size, you must experiment with several values. On a GPU with 24 GB of VRAM, a batch size of roughly 1,000 usually works well. If the batch size is set too high, the pipeline will abort with a runtime error ("Analysis was cancelled due to an error."), and the log will state that the GPU ran out of memory.

  • Run on three GPUs: birdnet-benchmark soundscape.wav --backend "pb" --worker 3 --device "GPU:0" "GPU:1" "GPU:2" --batch-size 1000

  • Increase amount of Producers: birdnet-benchmark soundscape.wav --producers 2

  • Increase Buffer size to 3 * Worker: birdnet-benchmark soundscape.wav --prefetch-ratio 2

File types within the run folder

Category Filename Contents
Runtime Statistics stats-{timestamp}.txt Summary of the key metrics for the run.
  stats-{timestamp}.json Complete metric set in JSON format.
Inference Results result-{timestamp}.npz Space-efficient binary file containing per-segment probabilities for all species (source for all other formats).
  result-{timestamp}.csv Tabular view of probabilities; first column holds the full recording path.
Log log-{timestamp}.log Full log of the benchmark run.

Cross-Run Overview – The parent directory maintains a file named runs.csv containing the metrics of all runs in chronological order, enabling comparative analyses. Example output on Linux:

Benchmark folder:
  /home/user/.local/share/birdnet/acoustic-benchmarks/v2.4/lib-v0.2.0a0/run-20250710T143348
Statistics results written to:
  /home/user/.local/share/birdnet/acoustic-benchmarks/v2.4/lib-v0.2.0a0/run-20250710T143348/stats-20250710T143348.txt
  /home/user/.local/share/birdnet/acoustic-benchmarks/v2.4/lib-v0.2.0a0/run-20250710T143348/stats-20250710T143348.json
  /home/user/.local/share/birdnet/acoustic-benchmarks/v2.4/lib-v0.2.0a0/runs.csv
Prediction results written to:
  /home/user/.local/share/birdnet/acoustic-benchmarks/v2.4/lib-v0.2.0a0/run-20250710T143348/result-20250710T143348.npz
  /home/user/.local/share/birdnet/acoustic-benchmarks/v2.4/lib-v0.2.0a0/run-20250710T143348/result-20250710T143348.csv
Log file written to:
  /home/user/.local/share/birdnet/acoustic-benchmarks/v2.4/lib-v0.2.0a0/run-20250710T143348/log-20250710T143348.log

Interpretation of runtime metrics

During analysis, performance indicators are updated and printed once per second.

Abbr. Meaning Target / Recommendation
SPEED Acceleration factor relative to real-time (RT). 2 xRT means ten minutes of audio are processed in five. Startup overhead and one-time model loading per process are excluded. Derived from mean worker runtime relative to processed audio duration. Also reports segments/sec. As high as possible; typically ≥ 50 xRT.
MEM Total main-memory usage of the Python parent process plus subprocesses, including shared memory (MB). Keep below available RAM.
BUF Average number of batches in the buffer, shown as current / maximum. For W workers: BUF ≈ 2W / 2W.
WAIT Mean waiting time (ms) that workers spend waiting for new batches. NVMe SSD: ≤ 1 ms.
BUSY Average number of simultaneously active workers. Ideally W / W.
PROG Overall analysis progress in percent. 0% → 100%.
ETA Estimated time to completion. As small as possible.

Example log line:

SPEED: 51 xRT [17 seg/s]; MEM: 1590 M; BUF: 8/8; WAIT: 0.17 ms; BUSY: 4/4; PROG: 93.5 %; ETA: 0:00:48

Typical bottlenecks and mitigation measures

  • High WAIT values or empty buffer – Increase the number of Producers. If insufficient, use faster storage (NVMe/SSD) or reduce Workers.
  • BUSY < Worker count – Typically an I/O bottleneck. Apply steps above.
  • Cache effect – OS file caching boosts SPEED significantly on the second pass. For benchmarking, use only runs starting from the second pass.

Metrics after completion

After analysis completes, the benchmark tool reports:

  • Total Execution Time (Wall Time) – Program start → completion.
  • Average Buffer Size (Buffer) – Mean number of batches in the working buffer.
  • Worker Utilisation (Busy Workers) – Average number of active workers; mean wait time shown in parentheses.
  • Memory Utilisation (Memory Usage) – Peak RAM consumption including buffer and result array.
  • Processing Throughput (Performance) – Most informative metric. Expressed as × real-time: cumulative audio hours divided by total execution time. Also shows segments/sec and audio-sec/sec.
  • Computational Performance (Worker Performance) – Final compute speed, identical to the final SPEED value.