Add Simpatico as the compression subsystem#937
Draft
joosthooz wants to merge 4 commits into
Draft
Conversation
simpatico_codegen uses nvrtc.h for runtime CUDA compilation. The pixi env previously only pulled in cuda-nvrtc (runtime SO); add cuda-nvrtc-dev to provide the header.
Integrate simpatico_codegen (from felipeblazing/simpatico@6ce5d43) as the compression subsystem at src/compression/simpatico_codegen/. CMake: add_subdirectory with EXCLUDE_FROM_ALL; SIMPATICO_BUILD_TESTS=ON registers simpatico tests alongside Sirius tests in ctest.
…ctually using this
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This copies in the sources of a project we've been working on called Simpatico.
It is a compression system aimed at tabular data, that can apply a tree of operations to the columns. Each operator can have multiple outputs (like RLE produces values and run counts), each of which can be fed into additional downstream operators.
These sources are basically a full rewrite of something @felipeblazing started, so that it is a C++ module/library with clear APIs. It has a code generation backend that will fuse kernels together to provide very good performance.
The original project mixed rust with C++ and had a lot of FFI and other boilerplate code, and each operator would transfer all data to/from DRAM (no fusion, no chunked smem caching). The exploration functionality however is still missing here, so we need to figure out if we really need it to proceed with this or if we can start with just some manually specified compression plans. It is written in Rust in the original repo so if we want to use that same implementation we will need to provide FFIs again (bi-directional, because Sirius needs to call the function but that code in turn needs to be able to call all the compressors to evaluate them).
This initial state does not integrate anything yet, but the plan file provides some clear directions for how to do that (with an agent) @kevkrist @bwyogatama.