[Feature]: Improve efficiency of `zstash check`

[forsyth2]

How will this affect the next version number?

New feature (increment MINOR version)

Is your feature request related to a problem?

Yes, @chengzhuzhang noted here:

There have been complaints about zstash check being slow.

Please note that this issue is similar, but not identical to #410. That issue was closed because the --tars option was a sufficient solution to the problem it considered:

We need zstash check to be able to immediately jump to a specified point

That is, that issue was resolved by running on fewer tars. This issue asks if we can get more tars-per-time-unit.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

Once #427 is merged, profiling should be added for zstash check as well. A few important considerations for that:

1. The existing performance records don't have check data, so we will need to allow for backwards compatibility in the performance scripts.
2. Add performance profiling infrastructure #427 already produces 5 relatively busy plots just for create, update, and extract (both sequential and parallel), so we will need to think about the best way to visualize check. Include it on the same output? A separate figure? Does it make sense to compare it to how long the full extract took?
3. Make sure that running zstash check doesn't interfere with other results. Considering we're running `extract twice (sequential and parallel), I think this should be feasible.

That is, this issue should only be considered resolved once we have both:

1. A more efficient zstash check
2. Performance profiling scripts that confirm this is so.

[forsyth2]

Also note that we may want to implement this in conjunction with #435/#445. Maybe we can address both maintainability & efficiency with a single refactoring.

[forsyth2]

Maybe we can address both maintainability & efficiency with a single refactoring.

I asked Claude in the same chat window as for #445 if we should do these issues together. It responds with the following. Most importantly, it notes these require orthogonal pieces of work. It does suggest doing the refactor (#435/#445) first -- but considering we just did a partial refactor (#416), we may want to consider addressing zstash check first.

---

Bottlenecks in zstash check

check() calls extract(keep_files=False), which runs the full extractFiles loop. With keep_files=False:

• Files are never written to disk
• But for every file: seek to offset, parse tar header, read the entire file contents in BLOCK_SIZE chunks, compute MD5, compare against the database

So the bottleneck is reading every byte of every tar and hashing it. The actual MD5 computation is fast; the limiting factor is almost certainly I/O throughput — either disk reads (if tars are cached locally) or HPSS/Globus download speed (if they need to be fetched first).

Two secondary bottlenecks worth noting:

• Sequential tar fetching: tars are fetched and checked one at a time in the single-worker path. The --workers flag exists but only parallelises the extraction/checking work, not the fetching — so if HPSS fetch latency is the real wall-clock cost, more workers don't help as much as they could.
• BLOCK_SIZE is 1024 × 1014 = ~1MB (note: that's 1014, not 1024 — looks like a typo). Not a major issue but slightly suboptimal for sequential reads.

The most meaningful speedup would come from pipelining fetch and check: while one tar is being checked, the next is being fetched in the background. That's a real architectural change.

---

Should this be part of the readability refactor?

No, and fairly strongly so. The performance work would touch extractFiles / multiprocess_extract / the HPSS fetch layer — which is largely orthogonal to the transfer-tracking and Globus-dispatch cleanup we just did. Mixing them would:

• Make the refactor harder to review ("is this change for readability or performance?")
• Make it harder to bisect if something breaks
• Mean the performance changes arrive without a clean baseline to benchmark against

The readability refactor also makes the performance work easier to do cleanly afterward — once extractFiles is less tangled, the pipelining change is easier to reason about and implement correctly. So the sequencing of "finish readability first, then performance" is genuinely the right order, not just a convenient excuse to defer things.

[forsyth2]

I had Copilot start a PR to improve zstash check efficiency: #447. So far, it's only touching zstash/extract.py, which is not 1 of the 4 currently changed files in #445, so we should be able to address these two issues independent of one another.

[forsyth2]

This actually appears to have more in common with #430 & #402, meaning perhaps the performance bottlenecks and the parallelism issues are related.