Description
Several bugs in the evaluation pipeline of
DeepLense_Classification_Transformers_Archil_Srivastava/ cause incorrect metrics
and prevent contributors from running evaluation.
Issues Found
1. micro_auroc is always NaN (eval.py:44, eval.py:68)
micro_auroc list is declared but never appended to. np.mean([]) silently
returns nan, which gets logged to W&B as the micro_auroc metric — completely
incorrect and silent.
2. softmax called without dim= (eval.py:172)
torch.nn.functional.softmax(metrics["logits"]) # missing dim=
PyTorch raises:
UserWarning: Implicit dimension choice for softmax has been deprecated.
Line 158 already does it correctly with dim=-1. Line 172 doesn't.
3. Hardcoded W&B entity (train.py:237, eval.py:96)
Any other contributor running this will either get an auth error or accidentally
log to the original author's W&B account. Should be configurable via CLI arg or
environment variable.
Proposed Fix
- Append micro_auroc values in the evaluation loop
- Add
dim=-1 to the softmax call on line 172
- Replace hardcoded entity with
--entity CLI argument defaulting to
os.environ.get("WANDB_ENTITY", None)
I'll submit a PR addressing all of these.
Description
Several bugs in the evaluation pipeline of
DeepLense_Classification_Transformers_Archil_Srivastava/cause incorrect metricsand prevent contributors from running evaluation.
Issues Found
1. micro_auroc is always NaN (eval.py:44, eval.py:68)
micro_auroclist is declared but never appended to.np.mean([])silentlyreturns
nan, which gets logged to W&B as themicro_aurocmetric — completelyincorrect and silent.
2. softmax called without dim= (eval.py:172)
PyTorch raises:
UserWarning: Implicit dimension choice for softmax has been deprecated.
Line 158 already does it correctly with
dim=-1. Line 172 doesn't.3. Hardcoded W&B entity (train.py:237, eval.py:96)
Any other contributor running this will either get an auth error or accidentally
log to the original author's W&B account. Should be configurable via CLI arg or
environment variable.
Proposed Fix
dim=-1to the softmax call on line 172--entityCLI argument defaulting toos.environ.get("WANDB_ENTITY", None)I'll submit a PR addressing all of these.