Skip to content

[GLUTEN-11885][VL] Respect custom VELOX_HOME in build info generation #11905

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
philo-he merged 4 commits into from
May 3, 2026
Merged

[GLUTEN-11885][VL] Respect custom VELOX_HOME in build info generation #11905

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
82af0f4
[GLUTEN-11885][VL] Respect custom VELOX_HOME in build info generation
officialasishkumar Apr 9, 2026
0b18892
Merge branch 'main' into fix/11885-build-info-velox-home-path
officialasishkumar Apr 14, 2026
a58483e
Merge remote-tracking branch 'origin/main' into fix/11885-build-info-…
officialasishkumar Apr 30, 2026
ee37515
[GLUTEN-11885][VL] Read typed and untyped Velox cache entries
officialasishkumar Apr 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions dev/builddeps-veloxbe.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,15 +120,15 @@ do
shift # Remove argument name from processing
;;
--velox_repo=*)
VELOX_REPO=("${arg#*=}")
VELOX_REPO="${arg#*=}"

@philo-he philo-he Apr 13, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we also need such modification for some other args?

@officialasishkumar officialasishkumar Apr 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the current script again, and I do not think we need the same change for the other args in this PR. The scalar handling here matters for --velox_repo, --velox_branch, and especially --velox_home because they are reused when building VELOX_PARAMETER and when propagating VELOX_HOME to child processes. The other flags are only consumed locally in this script today. If there is another arg that also needs to cross that boundary, I can handle it in a follow-up.

@philo-he philo-he Apr 14, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@officialasishkumar, thanks for the check. Since this functionality is not well covered by CI, could you verify it in local build with a custom Velox home used? You may simply check a local resource file generated by gluten-build-info.sh.

shift # Remove argument name from processing
;;
--velox_branch=*)
VELOX_BRANCH=("${arg#*=}")
VELOX_BRANCH="${arg#*=}"
shift # Remove argument name from processing
;;
--velox_home=*)
VELOX_HOME=("${arg#*=}")
VELOX_HOME="${arg#*=}"
shift # Remove argument name from processing
;;
--build_velox_tests=*)
Expand Down Expand Up @@ -208,6 +208,9 @@ fi

concat_velox_param

# Keep VELOX_HOME visible to child processes such as Maven's build-info generation step.
export VELOX_HOME

function build_arrow {
if [ ! -d "$VELOX_HOME" ]; then
get_velox
Expand Down
43 changes: 39 additions & 4 deletions dev/gluten-build-info.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ GLUTEN_ROOT=$(cd $(dirname -- $0)/..; pwd -P)

EXTRA_RESOURCE_DIR=$GLUTEN_ROOT/gluten-core/target/generated-resources
BUILD_INFO="$EXTRA_RESOURCE_DIR"/gluten-build-info.properties
BACKEND_HOME=""

# Delete old build-info file before regenerating
rm -f "$BUILD_INFO"
Expand All @@ -38,9 +39,9 @@ function echo_revision_info() {
function echo_velox_revision_info() {
BACKEND_HOME=$1
echo gcc_version=$(strings $GLUTEN_ROOT/cpp/build/releases/libgluten.so | grep "GCC:" | head -n 1)
echo velox_branch=$(git -C $BACKEND_HOME rev-parse --abbrev-ref HEAD)
echo velox_revision=$(git -C $BACKEND_HOME rev-parse HEAD)
echo velox_revision_time=$(git -C $BACKEND_HOME show -s --format=%ci HEAD)
echo velox_branch=$(git -C "$BACKEND_HOME" rev-parse --abbrev-ref HEAD)
echo velox_revision=$(git -C "$BACKEND_HOME" rev-parse HEAD)
echo velox_revision_time=$(git -C "$BACKEND_HOME" show -s --format=%ci HEAD)
}

function echo_clickhouse_revision_info() {
Expand All @@ -49,6 +50,35 @@ function echo_clickhouse_revision_info() {
echo ch_commit=$(cat $GLUTEN_ROOT/cpp-ch/clickhouse.version | grep -oP '(?<=^CH_COMMIT=).*')
}

function read_cmake_cache_value() {
CACHE_FILE="$GLUTEN_ROOT/cpp/build/CMakeCache.txt"
CACHE_KEY="$1"
if [ -f "$CACHE_FILE" ]; then
# Command-line -D values are stored as UNINITIALIZED unless CMake is given an explicit type.
grep "^${CACHE_KEY}:" "$CACHE_FILE" | cut -d= -f2- | head -n 1
fi
}

function resolve_velox_home() {
if [ -n "$BACKEND_HOME" ]; then
echo "$BACKEND_HOME"
return
fi

if [ -n "${VELOX_HOME:-}" ]; then
echo "$VELOX_HOME"
return
fi

CACHED_VELOX_HOME=$(read_cmake_cache_value VELOX_HOME)
if [ -n "$CACHED_VELOX_HOME" ]; then
echo "$CACHED_VELOX_HOME"
return
fi

echo "$GLUTEN_ROOT/ep/build-velox/build/velox_ep"
}

while (( "$#" )); do
echo "$1"
case $1 in
Expand All @@ -60,12 +90,17 @@ while (( "$#" )); do
echo backend_type="$BACKEND_TYPE" >> "$BUILD_INFO"
# Compute backend home path based on type
if [ "velox" = "$BACKEND_TYPE" ]; then
BACKEND_HOME="$GLUTEN_ROOT/ep/build-velox/build/velox_ep"
BACKEND_HOME=$(resolve_velox_home)
echo_velox_revision_info "$BACKEND_HOME" >> "$BUILD_INFO"
elif [ "ch" = "$BACKEND_TYPE" ] || [ "clickhouse" = "$BACKEND_TYPE" ]; then
echo_clickhouse_revision_info >> "$BUILD_INFO"
fi
;;
--backend_home|--velox_home)
if [ -n "$2" ]; then
BACKEND_HOME="$2"
fi
;;
--java)
echo java_version="$2" >> "$BUILD_INFO"
;;
Expand Down
Loading