Finalize benchmark test script

[jared321]

With these changes we should be able to compare MATLAB v3 against MATLAB v3 as well as Python v3 against v3. This should include using as references the original v3 benchmarks established for reviewing previous PRs.

While we are now able to create both MATLAB and Python v3 benchmarks, the comparison of those current benchmarks is not useful or interesting since the two sets of results are presently acquired with different nf_max values, which the script interprets as incompatible problem specifications. It will also fail to execute as expected because the integer value XYZ in the filename scheme _probXYZ refers to the row in dfo.dat and is 0-based in naming Python results; 1-based, for naming MATLAB results (ALL TO BE FIXED IN NEXT PR).

PR Self-review

• Try to break the code to confirm that all error handling works as expected.
• Since this PR effectively makes this test script official, perform mile-high review of all related code generated to support this. This included looking at all differences between the latest commit on this branch and the v0.1.0 release.
• Review all changes here
• Obtain new Python results at latest commit on this branch and confirm that they are identical to their respective v3 benchmarks acquired during the review of Establish final version (v3) of Python/POUNDERS benchmarks #297.
• Obtain new MATLAB results at latest commit on this branch and confirm that they are identical to their respective v3 benchmarks acquired during the review of Establish MATLAB v3 Benchmark Format #301.
• Confirm deterministic results with both MATLAB v3 and Python v3 obtained at latest commit on this branch
• Try to compare Python v3 and MATLAB v3 and confirm that they can't be compared since acquired with different nf_max
• Manually acquire new Python and MATLAB results at latest commit on branch with nf_max=500 for both, compare with test script, and confirm that they are reasonably and acceptably different.
  • This required making temporary changes to the Python benchmark test script to address the TODOs in that file (See above)
• Confirm all actions passing

[codecov]

Codecov Report

❌ Patch coverage is 8.33333% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.20%. Comparing base (abf917b) to head (5d19787).

Files with missing lines	Patch %	Lines
pounders/py/tests/compare_results.py	7.40%	25 Missing ⚠️
pounders/py/tests/load_results.py	11.11%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #302      +/-   ##
==========================================
+ Coverage   92.59%   93.20%   +0.60%     
==========================================
  Files         106      106              
  Lines        5011     4975      -36     
==========================================
- Hits         4640     4637       -3     
+ Misses        371      338      -33     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mmenickelly]

On a system that I anticipate actually using in the future (GCE), this script works as expected. Comparing two sets of benchmark results generated in sequence on the same machine by the same "extensive" benchmark test results in the benchmark test passing. Changing the data in one entry of the relevant data structure results in just that one comparison failing, resulting in an overall FAIL. This is exactly the behavior we expected.

My delay in this test was likely hardware-specific. My MacBook with an Intel chip was giving randomized results. Shockingly, the same test generated (at random) different comparisons failing when running the "extensive" benchmark test twice in a row. I could not diagnose this, and I am throwing away this MacBook at the end of the week (unrelated to this frustrating experience :-)). But since my testing is satisfactory on a machine I will continue to use, I will approve this PR.