Support Selective Metafield Population: Enable only _hoodie_commit_time for Incremental Reads

[suryaprasanna]

Context

Hudi currently supports an all-or-nothing approach for metafield population. Users can either:

• Populate all metafields (_hoodie_record_key, _hoodie_commit_time, _hoodie_partition_path, _hoodie_file_name, etc.)
• Disable all metafields to save on storage costs

When metafields are disabled, Hudi operations still function correctly because certain fields like _hoodie_record_key are virtualized - they're automatically generated on-the-go during operations like upserts. This virtualization approach works seamlessly without requiring the field to be physically stored.

However, incremental reads rely on the _hoodie_commit_time metafield to identify and read delta changes at the record level. This field is essential for incremental query patterns, enabling Hudi to provide efficient record-level change tracking.

Problem Statement

Users face a trade-off between storage efficiency and incremental read capabilities:

• Scenario 1: Users who disable metafields save storage but lose the ability to perform incremental reads because _hoodie_commit_time is not available
• Scenario 2: Users who enable metafields can perform incremental reads but pay for storing all metafields, even though most of them (like _hoodie_record_key, _hoodie_partition_path) can be virtualized and aren't strictly necessary

The gap: There's no option to selectively enable only the metafields that cannot be virtualized or are critical for certain query patterns (like _hoodie_commit_time for incremental reads) while excluding others that can be generated on-the-go.

This results in unnecessary storage overhead for users who need incremental reads but don't require the other metafields to be physically stored.

Proposed Solution

Introduce selective metafield population that allows users to:

• Enable only _hoodie_commit_time for incremental read support
• Nullify/disable other metafields (_hoodie_record_key, _hoodie_partition_path, _hoodie_file_name) that can be virtualized
• Achieve storage savings while maintaining incremental query capabilities

This would provide a middle ground between the current all-or-nothing approach, optimizing for both storage efficiency and functional requirements.

[danny0405]

+1, the gains are remarkable:

1. better read/write performance(eliminate unnecessary row transformation like appending metadata rows, also Ser/De IO);
2. less storage cost;
3. simpler read/write logic, we have many logic branches to handle the metadata fields specifically in read/write paths, which introduces unnecessary complexities.

[prashantwason]

Created an issue to track this: #18383

The issue includes a detailed design for a new config hoodie.meta.fields.to.exclude that allows selective meta field population. Excluded fields are written as null for optimal Parquet storage savings. Implementation covers all 4 write paths (Avro, Spark InternalRow, Spark SQL row-writer, Flink) with a pre-computed boolean[5] array for zero-overhead per-row checks.

[suryaprasanna]

Closing the discussion as we have an agreement on the change.