Splitting blocks larger than supported.#968
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
Prevents multinode/deepclust length-sorting from producing input blocks that exceed current representation limits (32-bit per-block sequence IDs and 40-bit packed offsets), and propagates failures cleanly to other workers instead of hanging.
Changes:
- Added
Cluster::Multinode::can_add_to_len_sorted_block()plus shared constants to centralize block limit checks for length-sorted inputs. - Updated
len_sort()to split/cap blocks using the centralized limit logic and to propagate errors via alensort_errormarker file. - Added a small unit-style check for boundary behaviors and wired it into CTest.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
src/test/test.cpp |
Adds a minimal unit-style test for can_add_to_len_sorted_block() boundary cases. |
src/cluster/multinode/len_sort.h |
Introduces shared constants and the can_add_to_len_sorted_block() declaration. |
src/cluster/multinode/len_sort.cpp |
Implements block-limit logic, updates len-sorting to respect limits, and adds error propagation to avoid worker hangs. |
CMakeLists.txt |
Registers a new unit test in CTest. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| add_test(NAME linclust COMMAND ${CMAKE_COMMAND} -DNAME=linclust "-DARGS=linclust -d ${TD}/nr_10k.faa -p4 --approx-id 0" ${SP}) | ||
| add_test(NAME linclust_reps COMMAND ${CMAKE_COMMAND} -DTEST_DIR=${CMAKE_SOURCE_DIR}/src/test -P ${CMAKE_SOURCE_DIR}/src/test/linclust_reps.cmake) | ||
| add_test(NAME realign COMMAND ${CMAKE_COMMAND} -DNAME=realign "-DARGS=realign -d ${TD}/nr_10k.faa -p1 --clusters ${TD}/linclust.out" ${SP}) | ||
| add_test(NAME unit COMMAND diamond test) |
|
|
||
| namespace Cluster { namespace Multinode { | ||
|
|
||
| constexpr uint64_t LEN_SORT_BLOCK_SEQ_LIMIT = (uint64_t)std::numeric_limits<BlockId>::max() - 1; |
Comment on lines
+257
to
+258
| if (seqs == 0) | ||
| throw runtime_error("Sequence exceeds supported maximum block size."); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Length-sorted blocks can exceed 4.3B sequences which causes overflow on the 32bit per-block sequence ids.
Summary
Prevent large-memory deepclust length sorting from creating input blocks that exceed 2^32-1 by creating additional input blocks when exceeding a limit. Currently for:
Details
Cluster::Multinode::can_add_to_len_sorted_block()to centralize length-sort block limit checks.input0.faa.lensort_errormarker to avoid hangs (just for bookkeeping to have all threads throw a clean error).Validation
cmake --build diamond/build --target diamond -j 128ctest --test-dir diamond/build --output-on-failure