Design decisions for updating parallelism in zstash

[forsyth2]

Updating parallelism in zstash

Temporary solution

#429 implements a temporary solution to just keep using fork. There are more robust solutions available, but they have proven to be very complex. My recommendation would be for us to merge #429 for the Spring Unified release, and then take some time to really evaluate the better parallelism workflows before the Fall Unified release.

Why is this not an ideal solution?

From Claude, when asked about the temporary solution:

If you explicitly set the multiprocessing start method to fork — something like multiprocessing.set_start_method("fork") near the top of the program — Python 3.14 would continue using the old clone-the-parent behavior, and none of the bugs described in this guide would have manifested. The SQLite cursor would be inherited, the old Condition/Queue objects would be inherited, HPSS credentials would be inherited. Everything would just work, the same as it did on 3.13.

So why not just do that? A few reasons it's not quite a free lunch:

• It's a workaround, not a fix. The Python core team switched the default to spawn for good reasons — fork is notoriously unsafe when the parent process uses threads, because forking a multithreaded process can cause deadlocks (a child might fork in the middle of another thread holding a lock, and that lock never gets released). Pinning to fork papers over this.
• It's not portable. fork is unavailable on Windows, so any future Windows support would still require the spawn-compatible fixes anyway.
• The spawn-compatible fixes are arguably better code. Explicitly opening per-worker DB connections and using manager-backed primitives is cleaner and more intentional than relying on silent inheritance.

So the two PRs did the "right" thing by making the code actually spawn-compatible, rather than opting out of the new default.

The second point is less relevant to us, but the first and third remain relevant -- at best to implement good software practices, and at worse to avoid crashing zstash.

Looking into the other solutions

My initial exploration into this didn't uncover the temporary solution above. First, I worked with Claude to implement #402. Later, I had Copilot implement #424. These ended up being two competing approaches.

During this exploration, I generated a large amount of documentation of the issues, in the form of GitHub comments:

• High-level summary
• Comparison chart of the 2 approaches
• Comparison diagrams of the 2 approaches
• Comparing performance of the 2 approaches
• Guide to parallelism in zstash (on the main branch)

Why are there so many PRs?

I'm learning how Copilot agents add work, which appears to almost always be in the form of new PRs, which ends up producing a large number of PRs quickly. Here's an explainer:

• Add Python 3.14 support #402 This is the first approach, generated largely with Claude.
  • Add baseline support for Python 3.14 #421 Copilot-generated PR that merges into Add Python 3.14 support #402's branch add-python3.14. I ended up cherry-picking the relevant commit into Add Python 3.14 support #402.
  • Add Python 3.14 support #425 Same as above
  • Add Python 3.14 support #428 Same as above. I haven't merged this one yet. It is simply Copilot addressing code review comments on Add Python 3.14 support #402.
  • [WIP] Add support for Python 3.14 #422 Copilot-generated PR that was created in error. (I meant to request a code review of Add Python 3.14 support #402)
• Fix Python 3.14 compatibility: AbsoluteLinkError, pickling errors, config propagation, HPSS hang, and print_all_contents deadlock in extract.py and parallel.py #424 Copilot-generated PR. This is the second approach. This merges into add-python3.14-try2, which is the same as Add Python 3.14 support #402's branch, but with an added commit addressing code-review comments specifcally on Python 3.14 support itself ("Address review comments")
• Fix Python 3.14+ parallelism: force fork start method and fix tar.extract filter #429 The temporary solution described at the top of this post
  • Add Python 3.14 support by forcing parallelism to use fork #434 The cleaned-up PR that will actually be merged to main.