When running AMLB in local mode in parallel (e.g., on a cluster or a large machine with a NFS). this code https://github.com/openml/automlbenchmark/blob/master/amlb/benchmark.py#L473 has a race condition in rare edge cases.
When two processes try to append to the same local file (session_dir/scores/results.csv) at the same time, one of the append operations gets dropped.
The resulting file will lock like this:
id,task,framework,constraint,fold,type,result,metric,mode,version,params,app_version,utc,duration,training_duration,predict_duration,models_count,seed,info,acc,balacc,logloss,models_ensemble_count
openml.org/t/2073,yeast,AutoGluonDefault14022025,4h8c,7,multiclass,-0.974113,neg_logloss,local,1.2.1b20250215,"","",2025-02-15T12:53:58,107.3,79.7,0.1,69,2031772959,,0.635135,0.509885,0.974113,4
openml.org/t/2073,yeast,AutoGluonDefault14022025,4h8c,9,multiclass,-1.07679,neg_logloss,local,1.2.1b20250215,"","",2025-02-15T12:54:02,111.4,82.1,0.1,69,818337949,,0.608108,0.556688,1.07679,5
But it should look like this:
id,task,framework,constraint,fold,type,result,metric,mode,version,params,app_version,utc,duration,training_duration,predict_duration,models_count,seed,info,acc,balacc,logloss,models_ensemble_count
openml.org/t/2073,yeast,AutoGluonDefault14022025,4h8c,7,multiclass,-0.974113,neg_logloss,local,1.2.1b20250215,"","",2025-02-15T12:53:58,107.3,79.7,0.1,69,2031772959,,0.635135,0.509885,0.974113,4
openml.org/t/2073,yeast,AutoGluonDefault14022025,4h8c,8,multiclass,-1.15063,neg_logloss,local,1.2.1b20250215,"","",2025-02-15T12:54:02,107.4,79.4,0.1,69,1120882699,,0.540541,0.47974,1.15063,4
openml.org/t/2073,yeast,AutoGluonDefault14022025,4h8c,9,multiclass,-1.07679,neg_logloss,local,1.2.1b20250215,"","",2025-02-15T12:54:02,111.4,82.1,0.1,69,818337949,,0.608108,0.556688,1.07679,5
The results will still be correctly saved to the global results file. Moreover, this does not crash the process. This would either require more unique session IDs or also a lock for the local results file.
When running AMLB in local mode in parallel (e.g., on a cluster or a large machine with a NFS). this code https://github.com/openml/automlbenchmark/blob/master/amlb/benchmark.py#L473 has a race condition in rare edge cases.
When two processes try to append to the same local file (
session_dir/scores/results.csv) at the same time, one of the append operations gets dropped.The resulting file will lock like this:
But it should look like this:
The results will still be correctly saved to the global results file. Moreover, this does not crash the process. This would either require more unique session IDs or also a lock for the local results file.