Dataset eth-sri/SWT-bench_Verified_bm25_27k_zsp produces low results when running the eval harness.

[juanmichelini]

Steps to reproduce:

1. Run inference on 10 instances of eth-sri/SWT-bench_Verified_bm25_27k_zsp.

2. Then run the eval harness with eth-sri/SWT-bench_Verified_bm25_27k_zsp, we got:

Total instances: 10
Submitted instances: 10
Resolved instances: 0
Unresolved instances: 9
Empty patch instances: 0
Error instances: 1
Eval limit: 10
Success rate: 0/10 (0.0%)

3. Then run the eval harness on the same inferred patches from before with princeton-nlp/SWE-bench_Verified, we got:

Total instances: 10
Instances completed: 10
Mean coverage: 0.8166666666666667
Mean coverage delta: 0.8166666666666667
Instances resolved: 9
Instances unresolved: 1
Instances with errors: 0
Instances still running: 0
Still existing images: 1

[nielstron]

Hi @juanmichelini,
I think what you are observing is expected behavior. SWT-bench_Verified_bm25_27k_zsp contains only the instances that passed our evaluation filters (described in the paper), e.g., only SWE-bench_Verified instances we could reproduce and that passed the flipped evaluation. If you run on the first 10 instances, there are likely different instances being evaluated. Our evaluation harness filters out the failed instances at the end, such that the golden solutions should still have 100% solve rate.