Profile multi process vs mutli thread

[shinzlet]

I heard somewhere that multiple processes would be better for our IO bandwidth - but I am having trouble finding a confident answer for that. Multithreading would be much easier to architect and aggregate status info from, so if its possible i think we should use that. Benchmarks should be done

[shinzlet]

From a little benchmark script (queues all the files in a small (100GB) ome zarr, about 100 files total, then copies them using worker threads / procs)

PS C:\Users\Daxi\Documents> uv run .\softcopy_benchmark.py
Enumerating files in N:\Seth\long9.ome.zarr …
Found 100 regular files
8 Workers

     Threads:    72964.1 MiB copied in  36.85s  →  1980.0 MiB/s
   Processes:    72964.1 MiB copied in  46.62s  →  1565.0 MiB/s
PS C:\Users\Daxi\Documents> uv run .\softcopy_benchmark.py
Enumerating files in N:\Seth\long9.ome.zarr …
Found 100 regular files
16 Workers

     Threads:    72964.1 MiB copied in  37.09s  →  1967.1 MiB/s
   Processes:    72964.1 MiB copied in  46.23s  →  1578.2 MiB/s
PS C:\Users\Daxi\Documents> 

So it looks like threads wins easily in real-world conditions. That's excellent :)