A lightweight, self-hostable tool for conducting subjective listening tests in a browser. Simply prepare your audio files and a YAML configuration file, and you're ready to run your experiment. Screenshots of the interface are shown below.
- Configuration-Driven: Define experiments in a single YAML file.
- Dual Backend Support: Choose the deployment option that fits your environment.
- PHP: Drop a static bundle into your
public_htmlorwwwdirectory. - FastAPI: Run it as a lightweight Python backend server.
- PHP: Drop a static bundle into your
- Built-in Analytics: Analyze experimental results directly in your browser without external plotting or statistical analysis tools.
- MOS: Rate each stimulus on a 1-5 scale.
- DMOS: Rate the degradation of each stimulus on a 1-5 scale.
- CMOS: Compare two stimuli and rate the degree of preference on a -3 to +3 scale.
- AB: Compare two stimuli and choose the preferred one.
- ABX: Determine whether the third sample (X) matches sample A or B.
- XAB: Determine whether the first sample (X) is closer to sample A or B.
- MUSHRA: Compare multiple stimuli and rate each on a 0-100 scale.
- Chrome (recommended)
- Edge
- Firefox
Safari is currently not supported due to issues with audio playback.
- Python 3.11+
- uv 0.9.17+
If you don't have uv, install it first.
curl -LsSf https://astral.sh/uv/install.sh | shThen, install the required dependencies by running:
make setupAn experiment is defined by a single YAML configuration file. The easiest way to get started is to copy one of the example configurations in examples/ and edit it. Each example is fully commented, including optional settings that are not covered in this README.
Here's what a typical MOS configuration looks like:
test_type: mos
title: "Audio Quality Evaluation"
instructions: |
Please listen to each audio sample carefully and rate its overall quality
on a scale from 1 (Bad) to 5 (Excellent).
You may replay each sample as many times as needed.
output:
format: csv
path: ./results/
stimuli_dirs:
# items_per_session: 1
systems:
- path: ./stimuli/examples/A
system: A
- path: ./stimuli/examples/B
# system: B # inferred from the basename of `path`Important
Filenames (excluding extension) must match across all system directories. Files that are missing from any directory are excluded from the experiment (a warning is shown when the configuration is loaded).
Tip
If you don't want every listener sitting through the entire stimulus set, items_per_session limits how many items each listener is presented with.
For example, setting it to 1 in a two-system experiment means each listener evaluates a single item (one pair of stimuli).
Choose this option if you already have PHP web hosting, such as a shared hosting service with FTP access but no shell access. Note that the bundle needs PHP 8.1+ on that server.
Start by running:
make export CONFIG=/path/to/your/config.yaml DEPLOY=/path/to/your/public_html/my-testThis creates a self-contained my-test/ directory containing your configuration, stimuli, and the required PHP scripts.
Open the experiment URL in your browser or share it with your participants.
To update the experiment, regenerate the bundle with
make export-force CONFIG=/path/to/your/another_config.yaml DEPLOY=/path/to/your/public_html/my-testThis replaces the exported files while preserving any results that have already been collected.
Important
Audio files are symlinked into the bundle instead of being copied, so they must be transferred separately when uploading or sharing the bundle.
To create a fully self-contained bundle, use make export-copy (or make export-copy-force to regenerate it).
This will copy the audio files directly into the bundle.
Choose this option if you can keep a Python process running, for example on an internal network or a small VM. Unlike the PHP deployment, the configuration is served directly, so no export step is required.
Start the server by running:
make serve CONFIG=/path/to/your/config.yamlThen open http://localhost:8000 in your browser to access the experiment.
To use a different host or port, specify them as Make variables:
make serve CONFIG=/path/to/your/config.yaml HOST=127.0.0.1 PORT=8080Results are stored in a subdirectory named after the configuration file, with one file per session:
results/
└── config/ # basename of your config file
├── 0009b4f1-b002-4ff2-998a-5694d7f916c5.csv
├── 12c23f42-3ade-4a43-9ff0-f1f6514b38df.csv
├── ...
For the PHP deployment, the results live inside the exported bundle. For example, you can generate a self-contained HTML report with:
make report CONFIG=/path/to/your/config.yaml DEPLOY=/path/to/your/public_html/my-testThis writes the report to /path/to/your/public_html/my-test/results/config/report.html in this case.
The report includes interactive Plotly visualizations, summary statistics, etc.
Open the generated HTML in your browser to view the report.
For the FastAPI deployment, simply run:
make report CONFIG=/path/to/your/config.yamlTo customize the report settings, specify a report configuration file:
make report CONFIG=/path/to/your/config.yaml REPORT_CONFIG=examples/report-config.yamlThe figure below is an example from a generated report.
Important
Randomization is independent across listeners. Trial and stimulus order are randomized independently for each listener. This tool does not use a Latin-square or any other counterbalanced design across sessions. If your study requires guaranteed counterbalancing, you will need to prepare the assignment yourself.
Important
Blinding is intended for honest participants. The browser never receives system names, file paths, or which stimulus the hidden reference matches. However, preventing a determined user from inspecting browser memory or network traffic is outside the scope of this tool.
Released under the 3-Clause BSD License. See LICENSE for details.