Skip to content

Move load_npy to C++ - #849

Merged
mthrok merged 2 commits into
mainfrom
npy-cpp
Jul 25, 2025
Merged

Move load_npy to C++#849
mthrok merged 2 commits into
mainfrom
npy-cpp

Conversation

@mthrok

@mthrok mthrok commented Jul 25, 2025

Copy link
Copy Markdown
Collaborator

So that the GIL is released entirely

Performance (based on data_formats.py)

  • QPS of loading NPY files with multi-threading
    Concurrency Baseline C++ Improvement
    32 1577 1891 17%
    16 1561 1717 9%
    8 1592 1859 14%
    4 1590 1812 12%
    2 1769 1820 3%
    1 1690 1948 13%
Screenshot 2025-07-25 at 4 59 08 PM
  • QPS of loading NPZ (no compression) files with multi-threading
    Concurrency Baseline C++ Improvement
    32 1577 1726 9%
    16 1495 1755 15%
    8 1607 1754 8%
    4 1591 1677 5%
    2 1637 1781 8%
    1 1658 1829 9%
Screenshot 2025-07-25 at 5 02 11 PM
  • QPS of loading NPZ (with compression) files with multi-threading
    Concurrency Baseline C++ Improvement
    32 1192 1473 19%
    16 1241 1633 24%
    8 1230 1640 25%
    4 1305 1548 16%
    2 1379 1547 11%
    1 1277 1571 19%
Screenshot 2025-07-25 at 5 05 32 PM

The performance of multiprocessing stays roughly same.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jul 25, 2025
@facebook-github-bot

Copy link
Copy Markdown
Contributor

@facebook-github-bot has imported this pull request. If you are a Meta employee, you can view this in D78947559. (Because this pull request was imported automatically, there will not be any future comments.)

@mthrok
mthrok force-pushed the npy-cpp branch 6 times, most recently from d49137c to f7c6015 Compare July 25, 2025 09:10
So that the GIL is released entirely
@mthrok mthrok changed the title Re-implement load_npy in C++ Move load_npy to C++ Jul 25, 2025
@mthrok
mthrok marked this pull request as ready for review July 25, 2025 09:34
@mthrok
mthrok merged commit 5d181f6 into main Jul 25, 2025
54 checks passed
@mthrok
mthrok deleted the npy-cpp branch July 25, 2025 11:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants