Gold Patch Failures/Errors (SWT-Bench Verified)

[akashgokul]

Describe the bug

Hi,

Thanks for your great work on this benchmark and repo.

I found that when running all the gold patches for Verified, after evaluation there are 12 instances that are still unresolved and 2 with errors. Note, I can run the small gold evaluation for sympy__sympy-20590 that is listed in the README, and it works as expected (1/1 resolved, no errors).

I looked at the logs and found that all the errors are environment related (e.g. module not found) and all errors happen in the pred_pre directory (pre-PR application).

Here is a table of the failure reasons for the unresolved/errored instances after running gold patch

Instance	Status	Exact Error Message
astropy__astropy-8872	error	ModuleNotFoundError: No module named 'setuptools.package_index'
pylint-dev__pylint-7277	error	ERROR: Project file:///testbed uses a build backend that is missing the 'build_editable' hook, so it cannot be installed in editable mode. Consider using a build backend that supports PEP 660.
pylint-dev__pylint-6386	unresolved	pylint.exceptions.UnknownMessageError: No such message id or symbol 'c-extension-no-member'.
pylint-dev__pylint-6903	unresolved	pylint.exceptions.UnknownMessageError: No such message id or symbol 'c-extension-no-member'.
pylint-dev__pylint-7080	unresolved	pylint.exceptions.UnknownMessageError: No such message id or symbol 'c-extension-no-member'.
sphinx-doc__sphinx-8459	unresolved	ModuleNotFoundError: No module named 'roman'
sphinx-doc__sphinx-8475	unresolved	ModuleNotFoundError: No module named 'roman'
sphinx-doc__sphinx-8548	unresolved	ModuleNotFoundError: No module named 'roman'
sphinx-doc__sphinx-8551	unresolved	ModuleNotFoundError: No module named 'roman'
sphinx-doc__sphinx-8593	unresolved	ModuleNotFoundError: No module named 'roman'
sphinx-doc__sphinx-8595	unresolved	ModuleNotFoundError: No module named 'roman'
sphinx-doc__sphinx-8621	unresolved	ModuleNotFoundError: No module named 'roman'
sphinx-doc__sphinx-8638	unresolved	ModuleNotFoundError: No module named 'roman'
sphinx-doc__sphinx-8721	unresolved	ModuleNotFoundError: No module named 'roman'

Steps/Code to Reproduce

Using the latest version of the codebase (release 1.3.0). Following installation instructions in README. Using python version 3.12.3

python -m src.main \
--dataset_name 'princeton-nlp/SWE-bench_Verified' \
--predictions_path gold \
--filter_swt \
--max_workers 24 \
--run_id validate-gold-full

Expected Results

Expecting 433/433 resolved instances. 0 unresolved instances. 0 errored instances.

Actual Results

12 unresolved and 2 errored instances.
So final performance should be 419/433

"unresolved_ids": [
"pylint-dev__pylint-6386",
"pylint-dev__pylint-6903",
"pylint-dev__pylint-7080",
"sphinx-doc__sphinx-8459",
"sphinx-doc__sphinx-8475",
"sphinx-doc__sphinx-8548",
"sphinx-doc__sphinx-8551",
"sphinx-doc__sphinx-8593",
"sphinx-doc__sphinx-8595",
"sphinx-doc__sphinx-8621",
"sphinx-doc__sphinx-8638",
"sphinx-doc__sphinx-8721"
],
"error_ids": [
"astropy__astropy-8872",
"pylint-dev__pylint-7277"
],

System Information

Python3.12.3
New venv created using the installation instructions in the repo readme

[nielstron]

Hey @akashgokul ,

Thanks for raising this issue. I can confirm that in particular the sphinx bug appears in the latest version of the repository when running the given command. Based on the inference traces of the leaderboard submissions, it must have appeared after 11th August (eOtter) and before 13th Sep (LogicStar), the traces for both OpenHands GPT-5 variants are inconclusive.

[akashgokul]

@nielstron Thanks for the quick response.

FWIW this sphinx bug seems to be an open issue with swe-bench too:SWE-bench/SWE-bench#461 (comment)