-
Notifications
You must be signed in to change notification settings - Fork 89
Add a protocol buffer decode kernel #4107
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
thirtiseven
wants to merge
165
commits into
NVIDIA:main
Choose a base branch
from
thirtiseven:protocol_buffer_jni_dev
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 4 commits
Commits
Show all changes
165 commits
Select commit
Hold shift + click to select a range
4aa7dbb
ai draft
thirtiseven a319d10
style
thirtiseven 458b583
address comments
thirtiseven 5c1bbf4
address comments
thirtiseven 0445daa
address comments
thirtiseven 8ddbf96
address comments
thirtiseven c9eea59
copyrights
thirtiseven f27fd6e
Merge branch 'main' into protocol_buffer_jni_dev
thirtiseven 617471d
update, added more tests
thirtiseven bdfac06
Merge branch 'protocol_buffer_jni_dev' of https://github.com/thirtise…
thirtiseven 3701cdf
style
thirtiseven 1bf7660
address comments
thirtiseven b012421
multi column processing
thirtiseven ecd3a38
fix merge
thirtiseven c86f78c
Support enum, required and default values
thirtiseven 3b37d28
style
thirtiseven 46d2130
Support nested types
thirtiseven 5f89e60
performance optimization
thirtiseven e311b4b
single-pass-kernel, with debug log, met unbreakable wall
thirtiseven c52782c
delete debug log
thirtiseven 47973a7
check point before schema projection option A, PROTOBUF_SINGLE_PASS=1…
thirtiseven d2595b1
Merge flat and nested api for from_protobuf
thirtiseven aa3c852
Fix nested type support
thirtiseven 7e0e77d
add a tests
thirtiseven ac39a4e
Kernal code clean up
thirtiseven fd7ec66
style change
thirtiseven 828e3c0
clean up code
thirtiseven ca5a921
address comments
thirtiseven edbfd98
ai self review and comment addressed
thirtiseven 34bcf0b
ai self review and comment addressed
thirtiseven 632448b
address comments
thirtiseven 46ef3b0
bug fixs
thirtiseven 5438a17
clean up code
thirtiseven c009f62
style
thirtiseven 6152733
address comments
thirtiseven 48204d5
clean up and split big files
thirtiseven 89e6e8c
style
thirtiseven f37d3c9
style
thirtiseven 78b1c60
address cc comments
thirtiseven 661f085
codex review and address
thirtiseven 8e09a47
gemini review and address
thirtiseven aa2586f
Merge branch 'main' into protocol_buffer_jni_dev
thirtiseven 3789cfc
adapt new api
thirtiseven 33ddacd
add micro benchmark
thirtiseven c56ed5b
merge reduce+scan, defer error check, combined occ scan
thirtiseven 2b113c7
O1 field number lookup in nested
thirtiseven ce408d6
Remove perf-results
thirtiseven 38348cc
Batched scalar extraction (2D grid kernel)
thirtiseven 3647299
style
thirtiseven 6af1577
address greptile comments
thirtiseven 0d9d105
address comments
thirtiseven e861f40
address greptile comments
thirtiseven 2ff75a9
address greptile comments
thirtiseven f6ebffe
address greptile comments
thirtiseven 7a312d4
address greptile comments
thirtiseven 4fa362c
address greptile comments
thirtiseven 54fda94
style
thirtiseven aeaf50c
address greptile comments
thirtiseven d50f089
address greptile comments
thirtiseven 63bc13a
address greptile comments
thirtiseven 8316f75
address greptile comments
thirtiseven 8ae9547
address greptile comments
thirtiseven 756010d
address greptile comments
thirtiseven 64de422
address greptile comments
thirtiseven aae6bca
address comments
thirtiseven 6e1c4d2
address greptile comments
thirtiseven 9c6dd70
address greptile comments
thirtiseven 6b2f494
address greptile comments
thirtiseven e40d5a7
Lookup table for repeated message child scan
thirtiseven 4b4f6f9
reflection refactor
thirtiseven 66daed1
reflection refactor
thirtiseven cd1763b
bug fixes
thirtiseven 0213721
comment address
thirtiseven 5357378
comment address
thirtiseven c4b1507
address comments
thirtiseven 3852f53
address comments
thirtiseven ce29c11
address comments
thirtiseven 96b20e0
address comments
thirtiseven 87c5e99
address comments
thirtiseven fca9ea7
address comments
thirtiseven 8e5473c
address comments
thirtiseven 95e2586
address comments
thirtiseven 67f2db8
address greptile comments
thirtiseven 06c15ee
address greptile comments
thirtiseven 14c6644
bugfix
thirtiseven 7a06053
address comments
thirtiseven 809daa1
nits
thirtiseven 3067f0b
address comments
thirtiseven 8aa0bdf
style
thirtiseven a13433f
style
thirtiseven e0cd7f5
fix bug
thirtiseven 5ef84e9
address comments
thirtiseven 4817d06
address comments
thirtiseven bdd2e40
address comments
thirtiseven 6101a4b
address human comments
thirtiseven 0213433
port back refactor from pr 0
thirtiseven 6bcac2f
style
thirtiseven ff83290
address comments from part0
thirtiseven 35b1cb8
nghia style self-check
thirtiseven 82bf02a
backport suggestions
thirtiseven fe73ada
address comments
thirtiseven d01a25b
copyright
thirtiseven 54223ca
fix
thirtiseven 70a214b
apply comments suggestions
thirtiseven 6e00831
apply refactor on headers
thirtiseven 1754032
Merge remote-tracking branch 'origin/main' into protocol_buffer_jni_dev
thirtiseven 02b7303
Merge origin/main into protocol_buffer_jni_dev
thirtiseven 8ea16b5
Remove stale mr parameter from call sites
thirtiseven 2e91499
Apply part0 brace style to dev-only code
thirtiseven c1f13c2
Replace copy_varlen_data_kernel with cub::DeviceMemcpy::Batched
thirtiseven 33e30f7
Remove unused threads/blocks variables
thirtiseven 7508a80
style
thirtiseven f428371
Propagate input null mask to output struct
thirtiseven 8d3ba07
Merge remote-tracking branch 'origin/main' into protocol_buffer_jni_dev
thirtiseven 262f0e7
Merge remote-tracking branch 'origin/main' into protocol_buffer_jni_dev
thirtiseven a6192f9
Merge remote-tracking branch 'origin/main' into protocol_buffer_jni_dev
thirtiseven 78b1a3e
review skill
thirtiseven 12d0295
address coemments
thirtiseven e4505ed
address coemments
thirtiseven 96a39b3
port comments address back
thirtiseven e6d8011
license
thirtiseven 363a514
Merge branch 'main' into protocol_buffer_jni_dev
thirtiseven bd45c2c
fix xfailed cases
thirtiseven e5918dc
Merge remote-tracking branch 'origin/main' into protocol_buffer_jni_dev
thirtiseven c9fdb69
style
thirtiseven 66e1e30
Merge remote-tracking branch 'origin/main' into protocol_buffer_jni_dev
thirtiseven 64a6b11
Merge remote-tracking branch 'origin/main' into protocol_buffer_jni_dev
thirtiseven 3d913b7
from_protobuf kernel (part3): nested struct decode with scalar children
thirtiseven 65e190e
local review changes
thirtiseven 3e1062d
address local comments
thirtiseven 9202f6e
address local comments
thirtiseven d36c2c7
Document cur non-advance in nested message scan kernel
thirtiseven fc15c88
Merge remote-tracking branch 'origin/main' into protobuf-jni-3b1-3b2-…
thirtiseven eca0de4
address comments
thirtiseven 4be3b4d
style
thirtiseven 53a82d3
Merge remote-tracking branch 'origin/main' into protobuf-jni-3b1-3b2-…
thirtiseven 82f13a6
address comments
thirtiseven 14c7869
address comments
thirtiseven 98f9335
name and format cleanup
thirtiseven a9ec912
Merge remote-tracking branch 'origin/main' into protobuf-jni-3b1-3b2-…
thirtiseven 93dd968
Merge branch 'protobuf-jni-3b1-3b2-nested-scalar' into protocol_buffe…
thirtiseven 9edb9e4
apply test refactor
thirtiseven f84a177
address follow up comments from test refactor
thirtiseven 4e1662e
part3 foolowup refactor
thirtiseven 56ae070
Merge remote-tracking branch 'origin/main' into protocol_buffer_jni_dev
thirtiseven 32b03d7
style
thirtiseven d22f2b0
Merge remote-tracking branch 'origin/main' into protocol_buffer_jni_dev
thirtiseven 2c6337a
fix after upmerge
thirtiseven 70a09a7
fix
thirtiseven c7831a8
style
thirtiseven a538fd0
Merge remote-tracking branch 'origin/main' into protocol_buffer_jni_dev
thirtiseven 43d3779
style
thirtiseven ce23c07
Merge remote-tracking branch 'origin/main' into protocol_buffer_jni_dev
thirtiseven f63b041
Merge remote-tracking branch 'origin/main' into protocol_buffer_jni_dev
thirtiseven 1042c32
refactor after 3b done
thirtiseven 0f63799
a big ai refactor
thirtiseven e51f0af
Fix proto2 singular message occurrence merging
thirtiseven 4d2776e
context related refactor
thirtiseven 2e7c7de
add test coverage and some clean up
thirtiseven e11af5e
Merge remote-tracking branch 'origin/main' into protocol_buffer_jni_dev
thirtiseven 5a3ca07
Merge remote-tracking branch 'origin/main' into protocol_buffer_jni_dev
thirtiseven 0329cee
apply new refactor
thirtiseven f952031
apply refactor of refactor back
thirtiseven 634309f
fix benchmark, delete duplicate tests
thirtiseven 793b8cc
fix
thirtiseven File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| /* | ||
| * Copyright (c) 2025, NVIDIA CORPORATION. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| #include "cudf_jni_apis.hpp" | ||
| #include "dtype_utils.hpp" | ||
| #include "protobuf_simple.hpp" | ||
|
|
||
| #include <cudf/column/column_view.hpp> | ||
|
|
||
| extern "C" { | ||
|
|
||
| JNIEXPORT jlong JNICALL | ||
| Java_com_nvidia_spark_rapids_jni_ProtobufSimple_decodeToStruct(JNIEnv* env, | ||
| jclass, | ||
| jlong binary_input_view, | ||
| jintArray field_numbers, | ||
| jintArray type_ids, | ||
| jintArray type_scales, | ||
| jboolean fail_on_errors) | ||
| { | ||
| JNI_NULL_CHECK(env, binary_input_view, "binary_input_view is null", 0); | ||
| JNI_NULL_CHECK(env, field_numbers, "field_numbers is null", 0); | ||
| JNI_NULL_CHECK(env, type_ids, "type_ids is null", 0); | ||
| JNI_NULL_CHECK(env, type_scales, "type_scales is null", 0); | ||
|
|
||
| JNI_TRY | ||
| { | ||
| cudf::jni::auto_set_device(env); | ||
| auto const* input = reinterpret_cast<cudf::column_view const*>(binary_input_view); | ||
| cudf::jni::native_jintArray n_field_numbers(env, field_numbers); | ||
| cudf::jni::native_jintArray n_type_ids(env, type_ids); | ||
| cudf::jni::native_jintArray n_type_scales(env, type_scales); | ||
| if (n_field_numbers.size() != n_type_ids.size() || | ||
| n_field_numbers.size() != n_type_scales.size()) { | ||
| JNI_THROW_NEW(env, | ||
| cudf::jni::ILLEGAL_ARG_EXCEPTION_CLASS, | ||
| "fieldNumbers/typeIds/typeScales must be the same length", | ||
| 0); | ||
| } | ||
|
|
||
| std::vector<int> field_nums(n_field_numbers.begin(), n_field_numbers.end()); | ||
| std::vector<int> encodings(n_type_scales.begin(), n_type_scales.end()); | ||
| std::vector<cudf::data_type> out_types; | ||
| out_types.reserve(n_type_ids.size()); | ||
| for (int i = 0; i < n_type_ids.size(); ++i) { | ||
| out_types.emplace_back(cudf::jni::make_data_type(n_type_ids[i], n_type_scales[i])); | ||
| } | ||
|
|
||
| auto result = spark_rapids_jni::decode_protobuf_simple_to_struct( | ||
| *input, field_nums, out_types, encodings, fail_on_errors); | ||
| return cudf::jni::release_as_jlong(result); | ||
| } | ||
| JNI_CATCH(env, 0); | ||
| } | ||
|
|
||
| } // extern "C" | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.