java - 构建 Apache Spark 时出错

Question

我已经成功使用 AmpCamp 的 Spark 版本来构建和运行几个项目。我现在尝试使用较新版本的 Spark 1.2.1，但似乎无法构建它。

环境:

$ mvn --version
Apache Maven 3.0.5
Maven home: /usr/share/maven
Java version: 1.8.0_31, vendor: Oracle Corporation
Java home: /usr/lib/jvm/jdk1.8.0_31/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.13.0-45-generic", arch: "amd64", family: "unix"

$ echo $JAVA_HOME
/usr/lib/jvm/jdk1.8.0_31

$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/jvm/jdk1.8.0_31/bin:/usr/lib/jvm/jdk1.8.0_31/db/bin:/usr/lib/jvm/jdk1.8.0_31/jre/bin:/usr/lib/jvm/jdk1.8.0_31/bin:/usr/lib/jvm/jdk1.8.0_31/db/bin:/usr/lib/jvm/jdk1.8.0_31/jre/bin

$ ant -version
Apache Ant(TM) version 1.9.3 compiled on April 8 2014

注意:尝试了“mvn -Pyarn -Phadoop-2.5 -Dhadoop.version=2.5.0 -Dscala-2.11 -X -DskipTests clean package”，结果如下所示。

$ mvn -DskipTests clean package

...
[INFO] --- maven-antrun-plugin:1.7:run (default) @ spark-core_2.10 ---
[WARNING] Parameter tasks is deprecated, use target instead
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Spark Project Parent POM .......................... SUCCESS [6.068s]
[INFO] Spark Project Networking .......................... SUCCESS [7.572s]
[INFO] Spark Project Shuffle Streaming Service ........... SUCCESS [5.036s]
[INFO] Spark Project Core ................................ FAILURE [1.919s]
[INFO] Spark Project Bagel ............................... SKIPPED
[INFO] Spark Project GraphX .............................. SKIPPED
[INFO] Spark Project Streaming ........................... SKIPPED
[INFO] Spark Project Catalyst ............................ SKIPPED
[INFO] Spark Project SQL ................................. SKIPPED
[INFO] Spark Project ML Library .......................... SKIPPED
[INFO] Spark Project Tools ............................... SKIPPED
[INFO] Spark Project Hive ................................ SKIPPED
[INFO] Spark Project REPL ................................ SKIPPED
[INFO] Spark Project Assembly ............................ SKIPPED
[INFO] Spark Project External Twitter .................... SKIPPED
[INFO] Spark Project External Flume Sink ................. SKIPPED
[INFO] Spark Project External Flume ...................... SKIPPED
[INFO] Spark Project External MQTT ....................... SKIPPED
[INFO] Spark Project External ZeroMQ ..................... SKIPPED
[INFO] Spark Project External Kafka ...................... SKIPPED
[INFO] Spark Project Examples ............................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 22.365s
[INFO] Finished at: Thu Feb 12 07:44:15 CST 2015
[INFO] Final Memory: 36M/536M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (default) on project spark-core_2.10: Error executing ant tasks: org/apache/tools/ant/launch/Locator: org.apache.tools.ant.launch.Locator -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :spark-core_2.10

我最初在 JAVA_HOME 中使用 JRE。阅读帖子Error building Spark with Maven ;尽管我在查找 javac 时没有错误，但遵循了相同的建议；但我仍然遇到同样的错误。

建议??

使用 -X 输出更新

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (default) on project spark-core_2.10: Error executing ant tasks: org/apache/tools/ant/launch/Locator: org.apache.tools.ant.launch.Locator -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (default) on project spark-core_2.10: Error executing ant tasks: org/apache/tools/ant/launch/Locator
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Error executing ant tasks: org/apache/tools/ant/launch/Locator
    at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:360)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
    ... 19 more
Caused by: java.lang.NoClassDefFoundError: org/apache/tools/ant/launch/Locator
    at org.apache.tools.ant.util.FileUtils.fromURI(FileUtils.java:1231)
    at org.apache.tools.ant.Location.<init>(Location.java:90)
    at org.apache.tools.ant.Location.<init>(Location.java:73)
    at org.apache.tools.ant.helper.ProjectHelper2$ProjectHandler.onStartElement(ProjectHelper2.java:729)
    at org.apache.tools.ant.helper.ProjectHelper2$RootHandler.startElement(ProjectHelper2.java:600)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXParser.java:509)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(XMLNSDocumentScannerImpl.java:379)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl$NSContentDriver.scanRootElementHook(XMLNSDocumentScannerImpl.java:605)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3138)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:880)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:649)
    at org.apache.tools.ant.helper.ProjectHelper2.parse(ProjectHelper2.java:307)
    at org.apache.tools.ant.helper.ProjectHelper2.parse(ProjectHelper2.java:178)
    at org.apache.tools.ant.ProjectHelper.configureProject(ProjectHelper.java:82)
    at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:264)
    ... 21 more
Caused by: java.lang.ClassNotFoundException: org.apache.tools.ant.launch.Locator
    at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass(ClassRealm.java:259)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:235)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:227)
    ... 43 more
[ERROR] 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :spark-core_2.10

Answer 1

我卸载了maven和ant。 安装了 Maven 并根据我们公司的配置配置了我的 settings.xml，现在我能够构建 Spark。也许我本地安装的 ant 导致了问题，或者我安装的 ant 版本与 pom.xml 文件中的内容不匹配。