[FEA][Story] Support ingest of Parquet VARIANT columns

[vuule]

Background

Apache Parquet's VARIANT logical type is the standard binary encoding for semi-structured data in the Parquet ecosystem. Spark 4.0, DuckDB, and parquet-mr all emit VARIANT-annotated columns when persisting JSON-like data, and Spark-RAPIDS already accepts VARIANT inputs. libcudf today does not recognize the VARIANT logical type at all -- a VARIANT column either fails to parse or is read as an opaque struct of binary children with no way to access its fields on the GPU.

This story tracks adding GPU-accelerated support for reading and extracting fields from Parquet VARIANT columns in libcudf and exposing it through pylibcudf, covering both the unshredded layout and Parquet's shredded VARIANT layout.

A VARIANT column is materialized in libcudf as struct<list<uint8> metadata, list<uint8> value, ...>. For unshredded inputs, only the metadata and value blobs are present. For shredded inputs, additional typed children (typed_value subtree) sit alongside the blobs and the per-row value lives in either value or one of the typed children depending on whether shredding succeeded for that row. Field extraction is a two-pass GPU kernel: a sizing pass walks the on-device parser to compute output lengths, then an offset scan and copy pass produces the result column. Shredding-aware extraction additionally consults the typed children before falling back to the binary blob.

Plan

Status	PR	Scope
✅ #22310	Add Parquet VARIANT reader infrastructure	Recognize the VARIANT logical type (union field 16) in the Thrift compact-protocol reader, route a VARIANT group's metadata/value BYTE_ARRAY children through the column buffer as list<uint8> via a new internal PARQUET_COLUMN_BUFFER_FLAG_VARIANT_BINARY, and guard sanitize_schema so it does not rewrite shredded VARIANT children. Reader is shredding-compatible: shredded files load and surface their typed children as ordinary cuDF columns. Pylibcudf shape tests on committed unshredded and shredded VARIANT fixtures.
✅ #22416	Core GPU variant field extraction (scalar + nested objects)	Public cudf::io::parquet::experimental::{get_variant_field, cast_variant, extract_variant_field} APIs that walk a JSONPath-like object-key path ($?(.name)+) on-device and decode the leaf value. Supports INT8/INT16/INT32/INT64 and STRING target types. Unshredded only.
✅ #22895	Array indexing	Extend the variant_path parser to accept the full JSONPath subset ($.foo[2].bar, $['weird.key']) and add name-step / index-step / quoted-key dispatch in the device walk. Inherits the unshredded-only behavior.
⏳ #TBD	Variant extraction microbenchmarks	NVBench divergence suite with dictionary-scan, field-scan, and found-vs-null scenarios for the kernels delivered in PRs 2-3.
⏳ #TBD	End-to-end example workload	Standalone cpp/examples/variant_workload/ exercising the read_parquet → extract_variant_field flow on representative VARIANT data, demonstrating the public API on realistic inputs.
⏳ #TBD	Multi-field extraction (issue #22897)	Batched get_variant_fields / extract_variant_fields entry points returning a cudf::table with one column per requested path. Faster than looping get_variant_field when paths share prefixes. Unshredded only.
⏳ #TBD	Shredding-aware variant extraction	Teach extract_variant_field (or add a parallel entry point) to honor Parquet's variant shredding fallback rules: per row, prefer the matching typed_value child when non-null, otherwise fall back to decoding value, otherwise treat as null/missing.

Out of scope (future stories)

• Variant write path -- producing VARIANT-annotated Parquet output. Today libcudf's parquet writer has no VARIANT case.
• extract_variant_field target types beyond INT{8,16,32,64} / STRING (FLOAT32/64, BOOL, DECIMAL, TIMESTAMP, nested → typed lists/structs).
• Variant filter pushdown -- using a path expression to skip row groups in the parquet reader before materialization.
• cudf-python and cudf-polars surface -- this story stops at pylibcudf. A follow-up story tracks cudf.DataFrame[col].variant.extract(path, dtype) accessors and a polars variant.field() operator.

References

• Apache Parquet VariantEncoding.md and VariantShredding.md
• Spark SPARK-44034 VARIANT data type
• DuckDB VARIANT support
• Apache parquet-testing VARIANT reference vectors