Parquet Hadoop fallback readVectored uses 128 KiB copies and can regress remote scans

[binmahone]

Summary

RAPIDS 26.06 regressed remote Parquet scan performance when reads used the Hadoop-backed RapidsInputFile.readVectored fallback. The regression was measured on S3A with spark.rapids.perfio.s3.enabled=false, but the same fallback is also used by GCS and other Hadoop-backed filesystems without an optimized readVectored override.

The fallback moved from the old Parquet copy loop, which used parquet.read.allocation.size with an 8 MiB default, to RapidsInputFile default readVectored, which delegated to HostMemoryBuffer.copyFromStream with a fixed 128 KiB internal copy chunk.

Initial Reproduction

The following were the initial single-run diagnostic measurements on NDS SF3000 query28, EMR on EKS, Spark 4.0.0, and six g6.4xlarge executors:

case	query28	scan buffer time
26.04.2 baseline	60-65 s	approximately 1.2-1.4k s
26.06.0 original	420.938 s	17112.416 s
26.06.0 + restored Hadoop fallback	72.400 s	1552.405 s
restored fallback + parquet.read.allocation.size=131072	434.112 s	17655.652 s

These measurements established that forcing a 128 KiB copy size was sufficient to reproduce the severe regression. They were not repeated or interleaved, so the 72.4 s fixed result versus the 60-65 s baseline range should not be interpreted as evidence of a stable residual 10% regression.

Follow-up Controlled Validation

The final implementation from NVIDIA/cudf-spark-jni#4765 and #15164 was compared with RAPIDS 26.04.2 using the same SparkAAS base image and resource settings. Five full NDS runs per side were interleaved as test1, baseline1, test2, baseline2, ... to reduce time-dependent environmental drift.

With PerfIO disabled, full-query-set times were:

pair	26.06 + fixes	26.04.2	delta
1	1034 s	1066 s	-3.00%
2	1024 s	1026 s	-0.19%
3	1029 s	991 s	+3.83%
4	1006 s	1000 s	+0.60%
5	975 s	992 s	-1.71%

The median paired delta was -0.19%. Mean times were 1013.6 s for 26.06 with the fixes and 1015.0 s for 26.04.2 (-0.14%). Independently computed medians were 1024 s and 1000 s (+2.40%), but they came from different pairs and did not agree with the paired median or mean.

Query28 itself remained noisy and changed direction across pairs:

pair	26.06 + fixes	26.04.2	delta
1	47.570 s	54.815 s	-13.22%
2	49.354 s	46.972 s	+5.07%
3	52.417 s	46.772 s	+12.07%
4	45.573 s	46.999 s	-3.03%
5	46.991 s	44.880 s	+4.70%

The independently computed query28 medians were 47.570 s and 46.972 s (+1.27%). The pair deltas ranged from -13.22% to +12.07%, so these runs did not reproduce a stable residual 10% regression.

As a control, the same full test was run five times per side with PerfIO enabled:

pair	26.06 + fixes	26.04.2	delta
1	750 s	756 s	-0.79%
2	767 s	763 s	+0.52%
3	771 s	782 s	-1.41%
4	778 s	778 s	0.00%
5	766 s	797 s	-3.89%

The median paired delta was -0.79%, and mean times were 766.4 s for 26.06 with the fixes and 775.2 s for 26.04.2 (-1.14%).

These results do not prove that every workload is regression-free, but they do not reproduce the initially suspected residual regression after the fix.

Code Path

Before #14674, GpuParquetScan.copyRemoteBlocksData used:

• PerfIO.readToHostMemory(...).getOrElse { inputFile.open(); copyDataRange(...) } for HadoopFileIO
• copyDataRange used parquet.read.allocation.size, defaulting to 8 MiB

After #14674, the fallback path used:

• inputFile.readVectored(...)
• RapidsInputFile default readVectored
• HostMemoryBuffer.copyFromStream(...)
• a fixed 128 KiB internal byte array

Fix

The fix has two layers and is being maintained on both release/26.06 and main:

Repository / layer	Target branch	Pull request	Status	Relationship
cudf-spark-jni	release/26.06	NVIDIA/cudf-spark-jni#4765	Merged	Original 26.06 fix
cudf-spark	release/26.06	#15164	Merged	Original 26.06 fix; uses the JNI helper from #4765
cudf-spark-jni	main	NVIDIA/cudf-spark-jni#4808	Open	Forward-port of #4765
cudf-spark	main	#15242	Open	Forward-port of #15164; uses the JNI helper from #4808

At the JNI layer, the default RapidsInputFile.readVectored fallback uses a bounded byte-buffer loop and exposes a caller-supplied-buffer helper. At the cudf-spark layer, HadoopInputFile.readVectored uses that helper with parquet.read.allocation.size, preserving Hadoop fallback configurability while keeping the #14674 inputFile.readVectored abstraction.

Optimized S3InputFile and IcebergS3InputFile implementations continue to override readVectored and use PerfIO/range-copy paths when enabled.

[binmahone]

Opened the fix as two PRs:

• Use larger copy buffer for vectored read fallback cudf-spark-jni#4765 updates the default RapidsInputFile.readVectored fallback to use a reusable 8 MiB copy buffer instead of HostMemoryBuffer.copyFromStream's 128 KiB chunks.
• Use configured copy buffer for Hadoop vectored reads #15164 overrides HadoopInputFile.readVectored to use the shared fallback helper with parquet.read.allocation.size.

The cudf-spark PR is draft because it depends on the JNI PR helper.

[tgravescs]

26.06.0 + restored Hadoop fallback time is still worse then 26.04.2 after your fix for hadoop fallback. how many times did you run? Still looks like 10% worse, do we know why?

[sameerz]

@binmahone does this affect more than S3A? If so we should update the issue description.

[binmahone]

26.06.0 + restored Hadoop fallback time is still worse then 26.04.2 after your fix for hadoop fallback. how many times did you run? Still looks like 10% worse, do we know why?

I also saw this 10%, but after a second run (even with #14674 reverted) my AI agent concluded that it's because of cache or jitter, so I didn't pay too much attention. Since you also observed this, I'll take another investigation.

[binmahone]

@binmahone does this affect more than S3A? If so we should update the issue description.

yes, in theory it will also affect gcs, I'll update issue description accordingly

[binmahone]

Follow-up on the apparent residual regression: the original 72.4 s fixed result and the 60-65 s baseline range were single-run diagnostic measurements, not repeated or time-interleaved measurements. They were useful for isolating the 128 KiB copy-size regression, but they were not sufficient to establish a remaining approximately 10% regression after the fix.

I reran the final implementation against 26.04.2 in five interleaved full-NDS pairs (test1, baseline1, test2, baseline2, ...) with the same SparkAAS base image and resource settings. With PerfIO disabled, the full-query-set pair deltas were -3.00%, -0.19%, +3.83%, +0.60%, and -1.71%. The paired-delta median was -0.19%; mean times were 1013.6 s for 26.06 with the fixes and 1015.0 s for 26.04.2 (-0.14%).

Query28 varied in both directions across the five pairs: -13.22%, +5.07%, +12.07%, -3.03%, and +4.70%. Its independently computed medians were 47.570 s and 46.972 s (+1.27%). This did not reproduce a stable residual 10% regression.

I preserved the original table in the issue as initial diagnostic evidence and added the controlled follow-up data and this qualification.

[yinqingh]

Ran NDS SF3K tests with current fix + PerfIO disabled on EC2 (6x g4dn.4xlarge) cc @binmahone

Total Time (s)

pair	26.06 + fixes	26.04.2	delta
run 1	1309.0	1378.0	-5.01%
run 2	1350.0	1393.0	-3.09%
run 3	1373.0	1337.0	+2.69%
run 4	1292.0	1332.0	-3.00%
run 5	1396.0	1321.0	+5.68%
average	1344.0	1352.2	-0.61%
median	1350.0	1337.0	+0.97%

[binmahone]

Both @yinqingh and my numbers indicate that there's no regression any more.

@gerashegalov please help to review this PR.

cc @sameerz @GaryShen2008