`test_loss` scale/behavior changed between v4.0.8 and latest (v4.2.1.1) when fine-tuning cpsam

[reinavi74]

Description

When fine-tuning the built-in cpsam model with the exact same training script, hyperparameters, and train/test data, test_loss values differ dramatically between Cellpose v4.0.8 and the latest release (v4.2.1.1) — even at epoch 0, before any gradient update (LR=0.000000), where the only thing being evaluated is the freshly-loaded pretrained checkpoint against the test set.

This suggests the change is not in model performance, but in how test_loss is computed/scaled internally, since the same checkpoint on the same test images produces very different reported values.

Environment

• Cellpose versions compared: 4.0.8 vs 4.2.1.1 (installed via pip)

Steps to reproduce

1. Load the built-in cpsam pretrained model.
2. Fine-tune via cellpose.train.train_seg() with identical arguments in both versions:
   • n_epochs=100, batch_size=1, learning_rate=1e-5, weight_decay=0.1
   • Same train_data/test_data (42 train images, 15 test images), same mask_filter, same seed.
3. Compare the logged train_loss/test_loss at each epoch, in particular epoch 0 (before any weight update).

Expected behavior

test_loss at epoch 0 (LR=0, i.e. no training has occurred yet) should be consistent across versions when evaluating the identical pretrained checkpoint on the identical test set — modulo minor floating point differences.

Actual behavior

test_loss at epoch 0 differs by roughly 11x between versions on the identical checkpoint and test data, and the newer version's test_loss stays in a narrow low band throughout training rather than showing the same relative improvement curve as the older version.

v4.0.8 (reproduced twice, in July and again in September on a fresh conda env — fully consistent):

0,  train_loss=3.2598, test_loss=1.5705, LR=0.000000
5,  train_loss=2.1895, test_loss=1.2801, LR=0.000006
10, train_loss=2.0329, test_loss=1.0054, LR=0.000010
20, train_loss=1.7770, test_loss=0.9517, LR=0.000010
30, train_loss=1.2932, test_loss=0.8421, LR=0.000010
40, train_loss=1.3691, test_loss=0.7821, LR=0.000010
50, train_loss=1.2276, test_loss=0.7407, LR=0.000005
60, train_loss=1.1041, test_loss=0.7384, LR=0.000001
70, train_loss=1.1146, test_loss=0.7381, LR=0.000000
80, train_loss=1.2890, test_loss=0.7382, LR=0.000000
90, train_loss=1.2523, test_loss=0.7384, LR=0.000000

Latest (v4.2.1.1), same script/data/hyperparameters:

0,  train_loss=2.0151, test_loss=0.1387, LR=0.000000
5,  train_loss=1.7474, test_loss=0.1488, LR=0.000006
10, train_loss=1.1042, test_loss=0.1519, LR=0.000010
20, train_loss=1.0503, test_loss=0.0934, LR=0.000010
30, train_loss=0.9962, test_loss=0.1483, LR=0.000010
40, train_loss=1.0254, test_loss=0.1182, LR=0.000010
50, train_loss=0.9191, test_loss=0.1054, LR=0.000005
60, train_loss=1.0240, test_loss=0.1118, LR=0.000001
70, train_loss=0.9876, test_loss=0.1200, LR=0.000000
80, train_loss=0.9551, test_loss=0.1198, LR=0.000000
90, train_loss=0.9512, test_loss=0.1206, LR=0.000000

Note also that train_loss trajectories are broadly comparable in shape/magnitude between the two versions (e.g. epoch 0: 3.26 vs 2.02; epoch 90: 1.25 vs 0.95), while test_loss is not (epoch 0: 1.57 vs 0.14) — suggesting the discrepancy is specific to the test-loss computation path rather than a general change in the loss function or model behavior.