Django log parser drops tests when multiple results appear on one line and also has some problems with the FAIL status

[jyyang7413]

Description

1. In SWT-Bench evaluations, Django test logs sometimes emit two (maybe multiple but so far I have only observed cases of two) test results on a single line. For example, while testing the gold predictions path with validate-gold as run_id, I get following line in run_instance_swt_logs/validate-gold/pred_pre__gold/django__django-10999/test_output.txt

test_negative (utils_tests.test_dateparse.DurationParseTests) ... test_parse_postgresql_format (utils_tests.test_dateparse.DurationParseTests) ... ok

The current parse_log_django in src/log_parsers.py uses a cross-line regex that only captures the first test name on the line and binds the trailing ok/FAIL to it. The second test is not corrrectly parsed, which later causes incorrect FAIL_TO_PASS / UNMATCHED classification.

Expected Behavior

• All test results on a line should be parsed.
• test_parse_postgresql_format (utils_tests.test_dateparse.DurationParseTests) should be recorded as PASSED.
• test_negative (utils_tests.test_dateparse.DurationParseTests) should be logged as FAILED by subsequent FAIL: summary lines in test_output.txt:

...
======================================================================
FAIL: test_negative (utils_tests.test_dateparse.DurationParseTests) (source='-15:30')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./tests/utils_tests/test_dateparse.py", line 125, in test_negative
    self.assertEqual(parse_duration(source), expected)
AssertionError: datetime.timedelta(-1, 85530) != datetime.timedelta(-1, 85470)
...

2. Additionally, the failed tests in django seem to be marked as FAIL, for example, in run_instance_swt_logs/validate-gold/pred_pre__gold/django__django-14376/test_output.txt I get following line:

test_options_non_deprecated_keys_preferred (dbshell.test_mysql.MySqlDbshellCommandTestCase) ... FAIL

However, parse_log_django currently uses FAILED instead of FAIL in django_suffixes to parse failed test, and this also causes subsequent ok lines to be swallowed or mis-parsed.

Suggested Fix

• Keep the cross-line regex for normal cases.
• Add a single-line multi-test handler:
  • If a line contains multiple test names and ends with ... ok/FAIL/ERROR/skipped, assign the status to the last test name on the line and leave status of the tests before (on the same line) to be decided by subsequent test output.
• Add FAIL to the status set so it is treated as TestStatus.FAILED.

Patch

diff --git a/src/log_parsers.py b/src/log_parsers.py
index 802073b..be169ad 100644
--- a/src/log_parsers.py
+++ b/src/log_parsers.py
@@ -66,9 +66,10 @@ django_suffixes = {
     "OK": TestStatus.PASSED.value,
     "skipped": TestStatus.SKIPPED.value,
     "FAILED": TestStatus.FAILED.value,
+    "FAIL": TestStatus.FAILED.value,
     "ERROR": TestStatus.ERROR.value,
 }
-django_pattern = re.compile(rf"(?P<testname>\b(test|app_unmigrated)([^\s]*)\s+\([^)]*\))((.|\n)(?!\.\.\.))*\s*\.\.\.((.|\n)(?!\b(test_|app_unmigrated|OK|ok|FAILED|ERROR|skipped)\b))*\s*\b(?P<status>OK|ok|FAILED|ERROR|skipped)\b", flags=re.MULTILINE)
+django_pattern = re.compile(rf"(?P<testname>\b(test|app_unmigrated)([^\s]*)\s+\([^)]*\))((.|\n)(?!\.\.\.))*\s*\.\.\.((.|\n)(?!\b(test_|app_unmigrated|OK|ok|FAILED|FAIL|ERROR|skipped)\b))*\s*\b(?P<status>OK|ok|FAILED|FAIL|ERROR|skipped)\b", flags=re.MULTILINE)
 django_fail_error_pattern = re.compile(rf"(?P<testname>\b(test_|app_unmigrated)([^\s]*)\s+\([^)]*\))")
 
 def parse_log_django(log: str) -> dict[str, str]:
@@ -81,13 +82,28 @@ def parse_log_django(log: str) -> dict[str, str]:
         dict: test case to test status mapping
     """
     test_status_map = {}
+    testname_pattern = re.compile(
+        r"\b(test|app_unmigrated)[^\s]*\s+\([^)]*\)"
+    )
+    inline_status_pattern = re.compile(r"\.\.\.\s+(?P<status>ok|OK|FAIL|ERROR|skipped)$")
 
-    # first try to extract normal run
+    # first try to extract normal run (cross-line matching)
     for match in django_pattern.finditer(log):
         test_name = match.group("testname")
         status = match.group("status")
         test_status_map[test_name] = django_suffixes[status]
 
+    # handle multiple tests on a single line (assign status to the last test)
+    for line in log.split("\n"):
+        status_match = inline_status_pattern.search(line.strip())
+        if not status_match:
+            continue
+        tests = [m.group(0) for m in testname_pattern.finditer(line)]
+        if len(tests) > 1:
+            test_status_map[tests[-1]] = django_suffixes[status_match.group("status")]
+            for test_name in tests[:-1]:
+                test_status_map.pop(test_name, None)
+
     # error and fail reports take precedence (this will overwrite any previous status)
     for line in log.split("\n"):
         for key, status in [("ERROR", TestStatus.ERROR), ("FAIL", TestStatus.FAILED)]:

However, above suggested fix might not be enough to handle all cases. In run_instance_swt_logs/validate-gold/pred_pre__gold/django__django-10554/test_output.txt, I get following output:

...
test_union_with_values_list_and_order (queries.test_qs_combinators.QuerySetSetOperationTests) ... test_union_with_values_list_on_annotated_and_unannotated (queries.test_qs_combinators.QuerySetSetOperationTests) ... ok
test_unsupported_intersection_raises_db_error (queries.test_qs_combinators.QuerySetSetOperationTests) ... skipped 'Database has feature(s) supports_select_intersection'
test_unsupported_ordering_slicing_raises_db_error (queries.test_qs_combinators.QuerySetSetOperationTests) ... ok

======================================================================
Destroying test database for alias 'default' ('file:memorydb_default?mode=memory&cache=shared')...
['--count', '-C', 'coverage.cover', '--include-pattern', '/testbed/(django/db/models/sql/query\\.py|django/db/models/sql/compiler\\.py)']
Testing against Django installed in '/testbed/django'
Importing application queries
Skipping setup of unused database(s): other.
Operations to perform:
  Synchronize unmigrated apps: auth, contenttypes, messages, queries, sessions, staticfiles
  Apply all migrations: admin, sites
Synchronizing apps without migrations:
  Creating tables...
    Creating table django_content_type
    Creating table auth_permission
    ... (omitted for brevity)
    Running deferred SQL...
Running migrations:
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying sites.0001_initial... OK
  Applying sites.0002_alter_domain_unique... OK
System check identified no issues (1 silenced).
Traceback (most recent call last):
  File "/root/trace.py", line 1119, in <module>
    main()
  File "/root/trace.py", line 1106, in main
    t.runctx(code, globs, globs)
  File "/root/trace.py", line 771, in runctx
    exec(cmd, globals, locals)
  File "./tests/runtests.py", line 564, in <module>
    options.start_at, options.start_after,
  File "./tests/runtests.py", line 310, in django_tests
    extra_tests=extra_tests,
  File "/testbed/django/test/runner.py", line 652, in run_tests
    result = self.run_suite(suite)
  File "/testbed/django/test/runner.py", line 594, in run_suite
    return runner.run(suite)
  File "/opt/miniconda3/envs/testbed/lib/python3.6/unittest/runner.py", line 183, in run
    result.printErrors()
  File "/opt/miniconda3/envs/testbed/lib/python3.6/unittest/runner.py", line 109, in printErrors
    self.printErrorList('ERROR', self.errors)
  File "/opt/miniconda3/envs/testbed/lib/python3.6/unittest/runner.py", line 115, in printErrorList
    self.stream.writeln("%s: %s" % (flavour,self.getDescription(test)))
  File "/opt/miniconda3/envs/testbed/lib/python3.6/unittest/runner.py", line 49, in getDescription
    return str(test)
  File "/opt/miniconda3/envs/testbed/lib/python3.6/unittest/case.py", line 1429, in __str__
    return "{} {}".format(self.test_case, self._subDescription())
  File "/opt/miniconda3/envs/testbed/lib/python3.6/unittest/case.py", line 1415, in _subDescription
    for (k, v) in sorted(self.params.items()))
  File "/opt/miniconda3/envs/testbed/lib/python3.6/unittest/case.py", line 1415, in <genexpr>
    for (k, v) in sorted(self.params.items()))
  File "/testbed/django/db/models/query.py", line 252, in __repr__
    data = list(self[:REPR_OUTPUT_SIZE + 1])
  File "/testbed/django/db/models/query.py", line 276, in __iter__
    self._fetch_all()
  File "/testbed/django/db/models/query.py", line 1240, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/testbed/django/db/models/query.py", line 184, in __iter__
    for row in compiler.results_iter(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size):
  File "/testbed/django/db/models/sql/compiler.py", line 1042, in results_iter
    results = self.execute_sql(MULTI, chunked_fetch=chunked_fetch, chunk_size=chunk_size)
  File "/testbed/django/db/models/sql/compiler.py", line 1077, in execute_sql
    sql, params = self.as_sql()
  File "/testbed/django/db/models/sql/compiler.py", line 475, in as_sql
    extra_select, order_by, group_by = self.pre_sql_setup()
  File "/testbed/django/db/models/sql/compiler.py", line 53, in pre_sql_setup
    order_by = self.get_order_by()
  File "/testbed/django/db/models/sql/compiler.py", line 359, in get_order_by
    raise DatabaseError('ORDER BY term does not match any column in the result set.')
django.db.utils.DatabaseError: ORDER BY term does not match any column in the result set.
...

There is no subsequent output that indicates the test state of test_union_with_values_list_and_order (queries.test_qs_combinators.QuerySetSetOperationTests). I guess when there are two tests on the same line, the status of former one is FAIL or ERROR.

Another Problem
I also came across following problem when I reevaluate e-otter's inference output on swt-bench verified with run_id validate-eotter. And I observe following output in run_instance_swt_logs/validate-eotter/pred_pre__claude_e_otter_plus_verfied/django__django-12308/test_output.txt:

...
test_prepopulated_fields (admin_views.tests.SeleniumTests) ... ['--count', '-C', 'coverage.cover', '--include-pattern', '/testbed/(django/contrib/admin/utils\\.py)']
Testing against Django installed in '/testbed/django'
Importing application admin_views
Skipping setup of unused database(s): other.
Operations to perform:
  Synchronize unmigrated apps: admin_views, auth, contenttypes, messages, sessions, staticfiles
  Apply all migrations: admin, sites
Synchronizing apps without migrations:
  Creating tables...
    Creating table django_content_type
    Creating table auth_permission
    ...(omitted for brevity)
    Running deferred SQL...
Running migrations:
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying sites.0001_initial... OK
  Applying sites.0002_alter_domain_unique... OK
System check identified no issues (0 silenced).
skipped 'No browsers specified.'
...

Current parse_log_django in src/log_parsers.py parses the status of test_prepopulated_fields (admin_views.tests.SeleniumTests) as PASSED, which actually is SKIPPED. The "Applying admin.0001_initial... OK" line in test output seems to interfere with the parse process.