[FEA] Support Databricks 18

[sameerz]

Is your feature request related to a problem? Please describe.
N/A

Describe the solution you'd like
Add support for Databricks 18 LTS

Describe alternatives you've considered
N/A

Additional context
https://docs.databricks.com/aws/en/release-notes/runtime/18

[nartal1]

From the release notes:
DBR 18 is built with:

• Apache Spark 4.1.0
• JDK 21 (Need to verify for cudf-spark)
• Delta Lake: 4.2.0 ([FEA] Support Delta Lake 4.2 release #14757)