java - 如何使用JDK 1.8将Hadoop AWS jar添加到Spark 2.4.5？

Question

我遇到了一个错误:java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found，偶然发现了有效的解决方案here。
但是，在答案后给出的注释中，作者指出以下内容:

com.amazonaws:aws-java-sdk-pom:1.11.760 : depends on jdk version hadoop:hadoop-aws:2.7.0: depends on your hadoop version s3.us-west-2.amazonaws.com: depends on your s3 location

因此，当我运行以下命令时:

pyspark --packages com.amazonaws:aws-java-sdk-pom:1.8.0_242,org.apache.hadoop:hadoop-aws:2.8.5

我遇到以下错误:

Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: com.amazonaws#aws-java-sdk-pom;1.8.0_242: not found]
    at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1302)
    at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:54)
    at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:304)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:774)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Traceback (most recent call last):
  File "/opt/app-root/lib/python3.6/site-packages/pyspark/python/pyspark/shell.py", line 38, in <module>
    SparkContext._ensure_initialized()
  File "/opt/app-root/lib/python3.6/site-packages/pyspark/context.py", line 316, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/opt/app-root/lib/python3.6/site-packages/pyspark/java_gateway.py", line 46, in launch_gateway
    return _launch_gateway(conf)
  File "/opt/app-root/lib/python3.6/site-packages/pyspark/java_gateway.py", line 108, in _launch_gateway
    raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number

我更改命令的原因如下:

JDK版本:

(app-root) java -version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)

Pyspark版本:2.4.5

Hadoop版本:2.8.5

如何解决此错误，并以正确的依赖项启动pyspark shell，以便从S3中读取文件？

Answer 1

这对我来说适用于spark:2.4.4-hadoop2.7:

    --conf spark.executor.extraClassPath=/hadoop-aws-2.7.3.jar:/aws-java-sdk-1.7.4.jar --driver-class-path /hadoop-aws-2.7.3.jar:/aws-java-sdk-1.7.4.jar