You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi team,
Scala writing to dataframe (9262) to csv taking 8.48 hours. We are seeing the process throwing OOO because of this long running and volume of the tables processed. Please let me know any way we can reduce performance for below line of code
here saveAsSingleFile=true
if (saveAsSingleFile) {
tableDF.coalesce(1).write.option("header", includeColumnNames).option("emptyValue",null).option("nullValue",null).mode(SaveMode.Overwrite).csv(pathToFolderWithCSV)
} else {
Hi team,
Scala writing to dataframe (9262) to csv taking 8.48 hours. We are seeing the process throwing OOO because of this long running and volume of the tables processed. Please let me know any way we can reduce performance for below line of code
here saveAsSingleFile=true
if (saveAsSingleFile) {
tableDF.coalesce(1).write.option("header", includeColumnNames).option("emptyValue",null).option("nullValue",null).mode(SaveMode.Overwrite).csv(pathToFolderWithCSV)
} else {