Skip to content

EVALUATION.md

Latest commit

 

History

History
119 lines (88 loc) · 5.22 KB

EVALUATION.md

File metadata and controls

119 lines (88 loc) · 5.22 KB
Raw

Evaluation

How to compute in-domain classification metrics and out-of-domain detection scores from a trained model.

Prerequisites

  • A trained model directory (exp/.../anti-spoofing_feat_model.pth + args.json).
  • The pre-encoded dataset the model was trained on (data/prepared_ds_seg_enc/).
  • For hierarchical models: the canonical label files committed at data/ (label_assignment.txt, label_assignment_superclass.txt, superclass_mapping_known.csv, superclass_mapping_test.csv). The path-resolution helper accepts either a repo-relative path or a name relative to --path_to_dataset.

Both eval scripts read args.json next to the model checkpoint to know what architecture to instantiate (flat / H-Shared / H-Arch) and how to scale logits (ArcFace vs sub-center).

In-domain classification

Script: scripts/get_classification_metrics.py.

uv run scripts/get_classification_metrics.py \
  --model_path exp/trained_models/<run_name>/anti-spoofing_feat_model.pth \
  --path_to_dataset data/prepared_ds_seg_enc \
  --superclass_lut data/superclass_mapping_known.csv \
  --batch_size 32

--superclass_lut accepts either a repo-relative path (as above) or a name resolved relative to --path_to_dataset. Required for hierarchical models; ignored for flat.

Or via makefile:

make id_eval

Output files (written next to the model):

exp/trained_models/<run_name>/
├── dev_in_domain_results.txt          # classification report on dev
├── eval_in_domain_results.txt         # classification report on eval (ID classes only)
├── dev_in_domain_results_sup.txt      # superclass report (hier only)
└── eval_in_domain_results_sup.txt     # superclass report (hier only)

The eval reports cover only samples whose model_name was in the train split — OOD samples are excluded here; they're scored by the OOD detector instead.

Out-of-domain detection

Script: scripts/ood_detector.py. Supports four scoring methods (--ood_method):

Method Description
mahalanobis Class-conditional Mahalanobis distance on penultimate features (default)
nsd Negative Stable Diffusion score (REFD baseline)
msp Maximum Softmax Probability
energy Energy-based score

For hierarchical models, --confidence_scaling selects how superclass and subclass scores are combined: local (subclass-only), sup (superclass-only), avg (average), none.

First run — compute features + score

The first invocation against a given model extracts and caches per-split features to <model_dir>/ood/{train,dev,eval}_dict.npy:

uv run scripts/ood_detector.py \
  --model_path exp/trained_models/<run_name>/anti-spoofing_feat_model.pth \
  --path_to_dataset data/prepared_ds_seg_enc \
  --label_assignment_file data/label_assignment.txt \
  --sup_label_assignment_file data/label_assignment_superclass.txt \
  --superclass_lut_known data/superclass_mapping_known.csv \
  --superclass_lut_full data/superclass_mapping_test.csv \
  --confidence_scaling sup \
  --ood_method mahalanobis \
  --batch_size 32

For flat models, the --sup_label_assignment_file and --superclass_lut_* arguments are accepted but not used. The --confidence_scaling argument is also a no-op.

Subsequent runs — reuse cached features

Once <model_dir>/ood/*.npy exists, add --ood_only to skip the slow feature-extraction step:

uv run scripts/ood_detector.py \
  --model_path exp/trained_models/<run_name>/anti-spoofing_feat_model.pth \
  --path_to_dataset data/prepared_ds_seg_enc \
  --label_assignment_file data/label_assignment.txt \
  --sup_label_assignment_file data/label_assignment_superclass.txt \
  --superclass_lut_known data/superclass_mapping_known.csv \
  --superclass_lut_full data/superclass_mapping_test.csv \
  --confidence_scaling sup \
  --ood_method nsd \
  --ood_only

The makefile target ood_eval is one example invocation:

make ood_eval

Output

exp/trained_models/<run_name>/ood/
├── train_dict.npy                          # cached features per split
├── dev_dict.npy
├── eval_dict.npy
├── OOD_eval_results_<method>.txt           # classification report on eval (ID vs OOD)
├── OOD_eval_results_sup_<method>.txt       # superclass-level report (hier only)
└── OOD_summary_<method>.json               # EER, threshold, AUC summary

The detector computes the EER threshold on dev (which must contain OOD examples), then applies it to eval and reports detection accuracy. If your dev split has no OOD samples, EER computation fails with All-NaN slice encountered.

Reading the metrics

  • EER (Equal Error Rate) — operating point where FPR = FNR. Lower is better. The detector picks the EER-optimal threshold on dev and reports both Dev EER and the resulting Eval EER using that threshold.
  • AUC-ROC — area under the OOD-score ROC. Higher is better.
  • Per-class precision/recall/F1 — in the printed classification reports, class 0 is ID and class 1 is OOD for OOD reports; for ID reports the class IDs match label_assignment.txt.
  • Superclass vs Global — for hierarchical models, Global is the leaf-class prediction (after combining superclass + subclass), Superclass is the parent-only prediction.