我写了一个字数统计代码,但是当我尝试使用下面的命令在 Windows 中从 CMD 运行它时,它抛出了一个异常。
spark-submit --class com.sample.WordCount --master local file:///E:/WordCountSample/target/WordCountSample-0.0.1-SNAPSHOT.jar file:///C:/Users/siddh/OneDrive/Desktop/sample.txt
pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>SparkSampleInScala</groupId>
<artifactId>WordCountSample</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>WordCountSample</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<!--<build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId> <version>3.0.0-M1</version>
</plugin> </plugins> </build> -->
<dependencies>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>2.4.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.scala-lang/scala-library -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.12.8</version>
</dependency>
</dependencies>
</project>
但是当我运行 spark submit 它抛出以下错误:
Exception in thread "main" java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/java8/JFunction2$mcIII$sp
at com.sample.WordCount$.main(WordCount.scala:22)
at com.sample.WordCount.main(WordCount.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NoClassDefFoundError: scala/runtime/java8/JFunction2$mcIII$sp
... 14 more
Caused by: java.lang.ClassNotFoundException: scala.runtime.java8.JFunction2$mcIII$sp
那么在这种情况下需要做什么?我正在使用 spark 2.4.0
版本和 scala 2.12.8 版本
,它们已经安装在我的 Windows 上。
请帮助我,因为几天以来我一直陷入困境。
提前致谢:)
最佳答案
看起来您正在使用 2.4.x 和 Scala 2.12。可能是兼容性问题。 Spark 文档引用:- Spark 在 Java 8+、Python 2.7+/3.4+ 和 R 3.1+ 上运行。对于 Scala API,Spark 2.4.0 使用 Scala 2.11。您将需要使用兼容的 Scala 版本 (2.11.x)。
关于java - spark提交抛出错误java.lang.ClassNotFoundException : scala. runtime.java8.JFunction2$mcIII$sp,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/54426075/