Skip to content

Use configured copy buffer for Hadoop vectored reads #15164

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
binmahone merged 2 commits into from
Jul 9, 2026
Merged

Use configured copy buffer for Hadoop vectored reads #15164

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ba4487c
Use configured copy buffer for Hadoop vectored reads
binmahone Jul 2, 2026
0453b56
Merge branch 'release/26.06' into fix-readvectored-hadoop-fallback
binmahone Jul 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 24 additions & 4 deletions sql-plugin/src/main/java/com/nvidia/spark/rapids/fileio/hadoop/HadoopInputFile.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025, NVIDIA CORPORATION.
* Copyright (c) 2025-2026, NVIDIA CORPORATION.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand All @@ -16,14 +16,15 @@

package com.nvidia.spark.rapids.fileio.hadoop;

import ai.rapids.cudf.HostMemoryBuffer;
import com.nvidia.spark.rapids.jni.fileio.RapidsInputFile;
import com.nvidia.spark.rapids.jni.fileio.SeekableInputStream;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

import java.io.IOException;
import java.util.List;
import java.util.Objects;
import java.util.OptionalLong;

Expand All @@ -34,21 +35,30 @@
* for reading the file.
*/
public class HadoopInputFile implements RapidsInputFile {
private static final String PARQUET_READ_ALLOCATION_SIZE = "parquet.read.allocation.size";

private final Path filePath;
private final FileSystem fs;
private final int copyBufferSize;

public static HadoopInputFile create(Path filePath, Configuration conf) throws IOException {
Objects.requireNonNull(filePath, "filePath can't be null!");
Objects.requireNonNull(conf, "Hadoop conf can't be null");
FileSystem fs = filePath.getFileSystem(conf);
return new HadoopInputFile(filePath, fs);
int copyBufferSize = conf.getInt(PARQUET_READ_ALLOCATION_SIZE,
RapidsInputFile.DEFAULT_READ_VECTORED_COPY_BUFFER_SIZE);
return new HadoopInputFile(filePath, fs, copyBufferSize);
Comment on lines +48 to +50

@greptile-apps greptile-apps Bot Jul 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Dependency on unpublished JNI artifact

Both RapidsInputFile.DEFAULT_READ_VECTORED_COPY_BUFFER_SIZE (line 49) and RapidsInputFile.readVectoredUsingCopyBuffer (line 91) are added in NVIDIA/cudf-spark-jni#4765, which has not yet been published in the 26.06 artifact. The PR description notes the two cannot find symbol compile errors are expected until that JNI change ships. This PR cannot be merged independently and must be gated on the JNI PR landing first.

}

private HadoopInputFile(Path filePath, FileSystem fs) {
private HadoopInputFile(Path filePath, FileSystem fs, int copyBufferSize) {
Objects.requireNonNull(filePath, "filePath can't be null!");
Objects.requireNonNull(fs, "FileSystem can't be null");
if (copyBufferSize <= 0) {
throw new IllegalArgumentException(PARQUET_READ_ALLOCATION_SIZE + " must be positive");
}
Comment on lines +56 to +58
this.filePath = filePath;
this.fs = fs;
this.copyBufferSize = copyBufferSize;
}

@Override
Expand All @@ -70,4 +80,14 @@ public OptionalLong getLastModificationTime() throws IOException {
public SeekableInputStream open() throws IOException {
return new HadoopInputStream(fs.open(filePath));
}

@Override
public void readVectored(HostMemoryBuffer output, List<RapidsInputFile.CopyRange> copyRanges)
throws IOException {
if (copyRanges.isEmpty()) {
return;
}
byte[] copyBuffer = new byte[copyBufferSize];

@gerashegalov gerashegalov Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking for a follow-up: HadoopInputFile could allocate min(copyBufferSize, largest requested range) like the JNI default path does, instead of always allocating copyBufferSize. That keeps the current per-call lifetime and thread-safety properties, while avoiding an 8 MiB allocation for small footer/range reads.

RapidsInputFile.readVectoredUsingCopyBuffer(this, output, copyRanges, copyBuffer);
}
Comment on lines +89 to +92
}
Loading