Describe the bug
When attempting to write data from a Delta table with a float column to a a table in Azure Data Explorer using the spark kusto connector, the following error occurs:
(com.microsoft.azure.kusto.data.exceptions.DataServiceException) DataType "float" is unsupported for entities of kind "Column". Reason="Column: Name="WarrantySaving" uses unsupported data type, DataType="float""
To Reproduce
Steps to reproduce the behavior:
pyKusto = SparkSession.builder.appName("kustoPySpark").getOrCreate()
dataframe_input.write.
format("com.microsoft.kusto.spark.datasource").
option("kustoCluster",connection_input["kustoCluster"]).
option("kustoDatabase",connection_input["kustoDatabase"]).
option("kustoTable", connection_input["kustoTable"]).
option("kustoAadAppId",connection_input["kustoAadAppId"]).
option("kustoAadAppSecret",connection_input["kustoAadAppSecret"]).
option("kustoAadAuthorityID",connection_input["kustoAadAuthorityID"]).
mode("Append").
save()
Expected behavior
The data is appended to the Kusto table without any error
Additional context
Spark cluster
com.microsoft.azure.kusto:kusto-spark_3.0_2.12:3.1.15 on 13.1 (includes Apache Spark 3.4.0, Scala 2.12)
Kusto cluster
Standard_L8s_v3
Describe the bug
When attempting to write data from a Delta table with a float column to a a table in Azure Data Explorer using the spark kusto connector, the following error occurs:
(com.microsoft.azure.kusto.data.exceptions.DataServiceException) DataType "float" is unsupported for entities of kind "Column". Reason="Column: Name="WarrantySaving" uses unsupported data type, DataType="float""
To Reproduce
Steps to reproduce the behavior:
pyKusto = SparkSession.builder.appName("kustoPySpark").getOrCreate()
dataframe_input.write.
format("com.microsoft.kusto.spark.datasource").
option("kustoCluster",connection_input["kustoCluster"]).
option("kustoDatabase",connection_input["kustoDatabase"]).
option("kustoTable", connection_input["kustoTable"]).
option("kustoAadAppId",connection_input["kustoAadAppId"]).
option("kustoAadAppSecret",connection_input["kustoAadAppSecret"]).
option("kustoAadAuthorityID",connection_input["kustoAadAuthorityID"]).
mode("Append").
save()
Expected behavior
The data is appended to the Kusto table without any error
Additional context
Spark cluster
com.microsoft.azure.kusto:kusto-spark_3.0_2.12:3.1.15 on 13.1 (includes Apache Spark 3.4.0, Scala 2.12)
Kusto cluster
Standard_L8s_v3