Spark and Hadoop Version confirmation

[boundless29]

Sorry for previously not asking the question on the right platform. I need clarity around exact version of Spark. Did you use spark2.4.7 with Hadoop 2.7?

I also noticed hadoopAWSVersion = 2.8.5. With what version of Hadoop this is compatible?

Thanks a lot for your quick support.

[sushil-thasale]

This application was tested with Spark 2.4.7 and Hadoop 2.8.5 on EMR 5.30.1. Also, it is using Scala 2.11.12.
If you plan on upgrading to Spark 3, I would recommend using Spark 3.1.2, Scala 2.12.10, Hadoop 3.2.1.

[MaheshReddy-battu]

@sushil-thasale @gw-vsasidharan

I have tried upgrading versions for spark & scala as above mentioned. But unfortunately there are build dependency errors .

Could not determine the dependencies of task ':shadowJar'.

Could not resolve all dependencies for configuration ':runtimeClasspath'.
Could not find org.scala-lang:scala-library:2.12.10.12.
Searched in the following locations:
- file:/C:/Users/.m2/repository/org/scala-lang/scala-library/2.12.10.12/scala-library-2.12.10.12.pom
- file:/C:/Users/.m2/repository/org/scala-lang/scala-library/2.12.10.12/scala-library-2.12.10.12.jar
- https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.12.10.12/scala-library-2.12.10.12.pom
- https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.12.10.12/scala-library-2.12.10.12.jar
Required by:
project :
Could not find org.apache.spark:spark-core_2.12.10:3.1.2.
Searched in the following locations:
- file:/C:/Users/.m2/repository/org/apache/spark/spark-core_2.12.10/3.1.2/spark-core_2.12.10-3.1.2.pom
- file:/C:/Users/.m2/repository/org/apache/spark/spark-core_2.12.10/3.1.2/spark-core_2.12.10-3.1.2.jar
- https://repo.maven.apache.org/maven2/org/apache/spark/spark-core_2.12.10/3.1.2/spark-core_2.12.10-3.1.2.pom
- https://repo.maven.apache.org/maven2/org/apache/spark/spark-core_2.12.10/3.1.2/spark-core_2.12.10-3.1.2.jar
Required by:
project :
Could not find org.apache.spark:spark-sql_2.12.10:3.1.2.
Searched in the following locations:
- file:/C:/Users/.m2/repository/org/apache/spark/spark-sql_2.12.10/3.1.2/spark-sql_2.12.10-3.1.2.pom
- file:/C:/Users/.m2/repository/org/apache/spark/spark-sql_2.12.10/3.1.2/spark-sql_2.12.10-3.1.2.jar
- https://repo.maven.apache.org/maven2/org/apache/spark/spark-sql_2.12.10/3.1.2/spark-sql_2.12.10-3.1.2.pom
- https://repo.maven.apache.org/maven2/org/apache/spark/spark-sql_2.12.10/3.1.2/spark-sql_2.12.10-3.1.2.jar
Required by:
project :
Could not find com.fasterxml.jackson.module:jackson-module-scala_2.12.10:2.9.9.
Searched in the following locations:
- file:/C:/Users/.m2/repository/com/fasterxml/jackson/module/jackson-module-scala_2.12.10/2.9.9/jackson-module-scala_2.12.10-2.
9.9.pom
- file:/C:/Users/.m2/repository/com/fasterxml/jackson/module/jackson-module-scala_2.12.10/2.9.9/jackson-module-scala_2.12.10-2.
9.9.jar
- https://repo.maven.apache.org/maven2/com/fasterxml/jackson/module/jackson-module-scala_2.12.10/2.9.9/jackson-module-scala_2.12.10-2.9.9.pom
- https://repo.maven.apache.org/maven2/com/fasterxml/jackson/module/jackson-module-scala_2.12.10/2.9.9/jackson-module-scala_2.12.10-2.9.9.jar
Required by:
project :
Could not find org.rogach:scallop_2.12.10:3.3.1.
Searched in the following locations:
- file:/C:/Users/.m2/repository/org/rogach/scallop_2.12.10/3.3.1/scallop_2.12.10-3.3.1.pom
- file:/C:/Users/.m2/repository/org/rogach/scallop_2.12.10/3.3.1/scallop_2.12.10-3.3.1.jar
- https://repo.maven.apache.org/maven2/org/rogach/scallop_2.12.10/3.3.1/scallop_2.12.10-3.3.1.pom
- https://repo.maven.apache.org/maven2/org/rogach/scallop_2.12.10/3.3.1/scallop_2.12.10-3.3.1.jar
Required by:
project :

• Try:
  Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

• Get more help at https://help.gradle.org

BUILD FAILED in 3s