Add Hudi table version 9 (Hudi 1.x) support to the Hudi target

[vinishjail97]

Background

Part of the Hudi 1.x upgrade (#762).

To keep #772 a clean binary upgrade, the Hudi target currently pins writes to table version 6 (the old timeline view) and disables auto-upgrade, so the write client does not upgrade tables to version 9:

• HudiTableManager.initializeHudiTable: .setTableVersion(HoodieTableVersion.SIX)
• HudiConversionTarget write config: .withWriteTableVersion(HoodieTableVersion.SIX.versionCode()) and .withAutoUpgradeVersion(false)

This preserves backward compatibility with the pre-upgrade behavior (instants selected/ordered by requested/instant time, column-stats index version V1).

Known limitation at table version 6 (index V1): decimal/fixed column stats

At column-stats index V1, Hudi's HoodieTableMetadataUtil.isColumnTypeSupported (the isColumnTypeSupportedV1 branch, when no record type is supplied) explicitly excludes DECIMAL, FIXED and BYTES columns ("DECIMAL's underlying type is BYTES"; tracked upstream as HUDI-8585). The same gate is used both by the MDT col-stats writer (getColumnsToIndex) and by XTable's parquet-footer fallback (HudiFileStatsExtractor.computeColumnStatsForFile).

Consequence on table version 6: a decimal/fixed column gets no column stats, even though the parquet footer contains valid min/max for it. Index V2 (table version 9) does support these types (isColumnTypeSupportedV2 does not exclude DECIMAL/FIXED/BYTES).

This is why TestHudiFileStatsExtractor currently asserts 8 of 9 columns (the decimal column is dropped) — all its tables/mocks are pinned to version 6.

Follow-up

Add table version 9 support to the Hudi target:

• Allow writing table version 9 (new MDT / timeline layout) behind a feature flag so users can opt in.
• Switch instant selection/ordering to completion time when on version 9, with tests covering out-of-order completion (see the discussion on HudiConversionSource).
• Parameterize the col-stats tests (e.g. TestHudiFileStatsExtractor) over table version 6 and 9: version 6 / index V1 -> decimal column dropped (8 columns); version 9 / index V2 -> decimal column present (9 columns). This will replace the current v6-only assertions.
• Column-stats index moves from V1 to V2 at table version 9; ensure the decimal/fixed value standardization paths are covered.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct