Commit cbae849
authored
Use configured copy buffer for Hadoop vectored reads (#15164)
Fixes #15163.
### Description
When an optimized input-file implementation is not selected, remote
Parquet reads use `HadoopInputFile` and the generic
`RapidsInputFile.readVectored` fallback. Before
NVIDIA/cudf-spark-jni#4765, that fallback used
`HostMemoryBuffer.copyFromStream` with a 128 KiB internal copy chunk,
while the pre-NVIDIA/cudf-spark#14674 Hadoop copy loop used
`parquet.read.allocation.size` with an 8 MiB default.
This PR overrides `HadoopInputFile.readVectored` to use the
caller-supplied-buffer helper from NVIDIA/cudf-spark-jni#4765:
- read the copy size from `parquet.read.allocation.size`
- default to the JNI fallback size of 8 MiB
- allocate a temporary buffer for each `readVectored` call
- avoid a retained shared buffer or `ThreadLocal` state
- leave `GpuParquetScan` on the `inputFile.readVectored` abstraction
introduced by #14674
The measured regression was on the S3A fallback with PerfIO disabled.
The same `HadoopInputFile` fallback is also used by GCS and other
Hadoop-backed filesystems that do not provide their own optimized
`readVectored` implementation. When S3 PerfIO is enabled,
`S3InputFile.readVectored` continues to use the optimized PerfIO path
and bypasses this fallback.
Requires the JNI change from NVIDIA/cudf-spark-jni#4765, merged as
`f5c95d88e16846a66a61870441e73c84863e706c`. The `release/26.06` base
branch selects JNI `26.06.1-SNAPSHOT` through #15228.
### Testing
The JNI change (`901ca4f570cb843c814121a7698ba8b3d5896fbf`) passed all
eight `RapidsInputFileTest` tests. After installing that JNI artifact
locally, the cudf-spark change commit
(`ba4487c750ba73cf2f2281288ca319ad7f3752be`) was packaged on an x86 host
with:
```text
mvn -B -pl sql-plugin -am -DskipTests \
-Dspark-rapids-jni.version=26.06.1-SNAPSHOT package
```
Result: `BUILD SUCCESS`.
### Performance Validation
Five interleaved full-NDS runs per configuration did not reproduce a
stable performance regression after the fix, with either PerfIO disabled
or enabled. See #15163 for the complete environment, per-run results,
and analysis.
### Checklists
Documentation
- [ ] Updated for new or modified user-facing features or behaviors
- [x] No user-facing change
Testing
- [ ] Added or modified tests to cover new code paths
- [ ] Covered by existing tests
- [x] Not required
Performance
- [ ] Tests ran and results are added in the PR description
- [x] Issue filed with a link in the PR description
- [ ] Not required
Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org>
Signed-off-by: Hongbin Ma <mahongbin@apache.org>1 parent 00e8528 commit cbae849
1 file changed
Lines changed: 24 additions & 4 deletions
Lines changed: 24 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | | - | |
| 2 | + | |
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
22 | | - | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| |||
34 | 35 | | |
35 | 36 | | |
36 | 37 | | |
| 38 | + | |
| 39 | + | |
37 | 40 | | |
38 | 41 | | |
| 42 | + | |
39 | 43 | | |
40 | 44 | | |
41 | 45 | | |
42 | 46 | | |
43 | 47 | | |
44 | | - | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
45 | 51 | | |
46 | 52 | | |
47 | | - | |
| 53 | + | |
48 | 54 | | |
49 | 55 | | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
50 | 59 | | |
51 | 60 | | |
| 61 | + | |
52 | 62 | | |
53 | 63 | | |
54 | 64 | | |
| |||
70 | 80 | | |
71 | 81 | | |
72 | 82 | | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
73 | 93 | | |
0 commit comments