基本上，我需要在 EMR 上运行烫金作业。同样的工作在我的 macbook 上的本地 hadoop 上运行得很好，但在 EMR 上的 Hadoop 上运行失败。
我也在 cascading-user 和 scala-user 组中努力寻求有关此问题的帮助，但未能成功。到目前为止，在过去几天尝试各种更改后，我没有取得太大进展。
这是我深入研究细节之前的错误:
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; at com.aggregation.job.DataAggregation$.(DataAggregation.scala:30) at com.aggregation.job.DataAggregation$.(DataAggregation.scala) at com.aggregation.job.DataAggregation.main(DataAggregation.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
他们说这可能是与二进制文件的 Scala 冲突，但我看不出任何明显的迹象。如果有人可以帮助解决这个问题，那就太好了。
环境:
Amazon EMR AMI:3.8.0(包括 Scala 2.11.1、Hadoop 2.4.0、Java 1.7.0_76 - AMI details here)
应用环境:Scalding 0.15.0、Scala 2.11.1、Java 1.7.0_80、Hadoop 2.4.0
我已尝试对配置进行各种更改，甚至在 EMR 中手动安装更高版本的 Scala，但到目前为止错误并没有消失。
求助!谢谢。
大号
设置
构建.sbt:
scalaVersion := "2.11.1"
ivyScala := ivyScala.value map {
_.copy(overrideScalaVersion = true)
}
依赖关系.sbt:
import AssemblyKeys._
val hadoopVersion = "2.4.0"
val scaldingVersion = "0.15.0"
libraryDependencies ++= Seq(
"com.twitter" %% "scalding-core" % scaldingVersion,
"com.twitter" %% "scalding-json" % scaldingVersion,
"com.twitter" %% "scalding-jdbc" % scaldingVersion,
"com.github.nscala-time" %% "nscala-time" % "2.0.0",
"org.apache.hadoop" % "hadoop-common" % hadoopVersion % "provided",
"org.apache.hadoop" % "hadoop-mapreduce-client-core" % hadoopVersion % "provided"
)
excludedJars in assembly <<= (fullClasspath in assembly) map { cp =>
val excludes = Set(
"jsp-api-2.1-6.1.14.jar",
"jsp-2.1-6.1.14.jar",
"jasper-compiler-5.5.12.jar",
"minlog-1.2.jar", // Otherwise causes conflicts with Kyro (which bundles it)
"janino-2.5.16.jar", // Janino includes a broken signature, and is not needed anyway
"commons-beanutils-core-1.8.0.jar", // Clash with each other and with commons-collections
"commons-beanutils-1.7.0.jar", // "
"hadoop-core-1.2.1.jar", // Provided by Amazon EMR. Delete this line if you're not on EMR
"hadoop-tools-1.2.1.jar" // "
)
cp filter { jar => excludes(jar.data.getName) }
}
resolvers ++= Seq(
"Conjars repo" at "http://conjars.org/repo"
)
程序集.sbt
import AssemblyKeys._
assemblySettings
mergeStrategy in assembly := Merge.mergeStrategy
项目/build.properties:
sbt.version=0.13.1
项目/程序集.sbt:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.0")
最后，这是使用 sbt-dependency-graph 的依赖树插件看看我是否有正确的版本。抱歉，太长了，如果我应该以其他方式显示此信息，请提出建议。
sbt 依赖树:
[info] com.abc.aggregator_2.11:0.1-20150628T184441 [S]
[info] +-com.github.nscala-time:nscala-time_2.11:2.0.0 [S]
[info] | +-joda-time:joda-time:2.7
[info] | +-org.joda:joda-convert:1.2
[info] |
[info] +-com.twitter:scalding-core_2.11:0.15.0 [S]
[info] | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-local:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | | +-org.codehaus.janino:janino:2.7.5
[info] | | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | | |
[info] | | | +-riffle:riffle:0.1-dev
[info] | | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | | |
[info] | | +-com.google.guava:guava:14.0.1 (evicted by: 15.0)
[info] | | +-com.google.guava:guava:15.0
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | | +-org.slf4j:slf4j-api:1.7.2 (evicted by: 1.6.6)
[info] | |
[info] | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | |
[info] | +-com.twitter:bijection-core_2.11:0.8.0 [S]
[info] | +-com.twitter:chill-algebird_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:algebird-core_2.11:0.10.0 (evicted by: 0.10.1)
[info] | | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | | |
[info] | | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill-hadoop:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | |
[info] | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:maple:0.15.0
[info] | | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-com.twitter:scalding-args_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-date_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-serialization_2.11:0.15.0 [S]
[info] | +-org.slf4j:slf4j-api:1.6.6
[info] |
[info] +-com.twitter:scalding-jdbc_2.11:0.15.0 [S]
[info] | +-cascading:cascading-jdbc-core:2.6.0
[info] | | +-com.google.guava:guava:15.0
[info] | |
[info] | +-cascading:cascading-jdbc-mysql:2.6.0
[info] | | +-cascading:cascading-jdbc-core:2.6.0
[info] | | | +-com.google.guava:guava:15.0
[info] | | |
[info] | | +-com.google.guava:guava:15.0
[info] | | +-mysql:mysql-connector-java:5.1.25
[info] | |
[info] | +-com.twitter:scalding-core_2.11:0.15.0 [S]
[info] | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-local:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | | +-org.codehaus.janino:janino:2.7.5
[info] | | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | | |
[info] | | | +-riffle:riffle:0.1-dev
[info] | | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | | |
[info] | | +-com.google.guava:guava:14.0.1 (evicted by: 15.0)
[info] | | +-com.google.guava:guava:15.0
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | | +-org.slf4j:slf4j-api:1.7.2 (evicted by: 1.6.6)
[info] | |
[info] | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | |
[info] | +-com.twitter:bijection-core_2.11:0.8.0 [S]
[info] | +-com.twitter:chill-algebird_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:algebird-core_2.11:0.10.0 (evicted by: 0.10.1)
[info] | | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | | |
[info] | | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill-hadoop:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | |
[info] | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:maple:0.15.0
[info] | | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-com.twitter:scalding-args_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-date_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-serialization_2.11:0.15.0 [S]
[info] | +-org.slf4j:slf4j-api:1.6.6
[info] |
[info] +-com.twitter:scalding-json_2.11:0.15.0 [S]
[info] +-com.fasterxml.jackson.module:jackson-module-scala_2.11:2.4.2 [S]
[info] | +-com.fasterxml.jackson.core:jackson-annotations:2.4.2
[info] | +-com.fasterxml.jackson.core:jackson-core:2.4.2
[info] | +-com.fasterxml.jackson.core:jackson-databind:2.4.2
[info] | | +-com.fasterxml.jackson.core:jackson-annotations:2.4.0 (evicted by: 2.4.2)
[info] | | +-com.fasterxml.jackson.core:jackson-annotations:2.4.2
[info] | | +-com.fasterxml.jackson.core:jackson-core:2.4.2
[info] | |
[info] | +-com.google.code.findbugs:jsr305:2.0.1
[info] | +-com.google.guava:guava:15.0
[info] | +-com.thoughtworks.paranamer:paranamer:2.6
[info] | +-org.scala-lang:scala-reflect:2.11.2 [S]
[info] |
[info] +-com.twitter:scalding-core_2.11:0.15.0 [S]
[info] | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-local:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | | +-org.codehaus.janino:janino:2.7.5
[info] | | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | | |
[info] | | | +-riffle:riffle:0.1-dev
[info] | | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | | |
[info] | | +-com.google.guava:guava:14.0.1 (evicted by: 15.0)
[info] | | +-com.google.guava:guava:15.0
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | | +-org.slf4j:slf4j-api:1.7.2 (evicted by: 1.6.6)
[info] | |
[info] | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | |
[info] | +-com.twitter:bijection-core_2.11:0.8.0 [S]
[info] | +-com.twitter:chill-algebird_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:algebird-core_2.11:0.10.0 (evicted by: 0.10.1)
[info] | | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | | |
[info] | | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill-hadoop:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | |
[info] | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:maple:0.15.0
[info] | | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-com.twitter:scalding-args_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-date_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-serialization_2.11:0.15.0 [S]
[info] | +-org.slf4j:slf4j-api:1.6.6
[info] |
[info] +-org.json4s:json4s-native_2.11:3.2.11 [S]
[info] +-org.json4s:json4s-core_2.11:3.2.11 [S]
[info] +-com.thoughtworks.paranamer:paranamer:2.6
[info] +-org.json4s:json4s-ast_2.11:3.2.11 [S]
[info] +-org.scala-lang:scalap:2.11.0
[info] +-org.scala-lang:scala-compiler:2.11.1 [S]
[info] +-org.scala-lang.modules:scala-parser-combinators_2.11:1.0.1 [S]
[info] +-org.scala-lang.modules:scala-xml_2.11:1.0.2 [S]
[info] +-org.scala-lang:scala-reflect:2.11.1 (evicted by: 2.11.2)
[info] +-org.scala-lang:scala-reflect:2.11.2 [S]
[info]
[success] Total time: 12 s, completed Jun 28, 2015 11:44:54 AM
根据要求提供更多信息:
我使用“sbt assembly”构建 fat jar，目前我正在使用带有“自定义 JAR”步骤的 AWS 控制台在自动化流程之前对其进行测试。
JAR location: s3://path/to/jar/data-aggregator-0.1.jar
Arguments: com.abc.aggregation.job.DataAggregation --hdfs --input s3n://path/to/input/data/file.json --output s3n://path/to/input/data/file.txt
更新:
通过提供指向 scala 2.11.1 jar 的 HADOOP_CLASSPATH，同时从 sbt 组装步骤中排除相同的内容，我能够克服上述错误。这是使用 hadoop-user-env.sh 传入的，似乎适用于主节点。然而，一旦它到达映射器步骤，它再次因另一个 Scala 错误而失败。现在我卡在了这一步。
假设这是因为映射器和缩减器没有看到 HADOOP_CLASSPATH 更新，我尝试包含指向 hadoop master 本身上的 scala jar 文件的 -libjars 参数。但这(下)似乎不起作用。
JAR location: s3://path/to/jar/data-aggregator-0.1.jar
Arguments: com.abc.aggregation.job.DataAggregation -libjars /usr/share/scala/lib/scala-library.jar,/usr/share/scala/lib/scala-reflect.jar --hdfs --input s3n://path/to/input/data/file.json --output s3n://path/to/input/data/file.txt
最佳答案
已修复。所以确实发生了在 EMR 实例中有多个 Scala jar，它们不是来 self 的应用程序 jar。
除了/usr/share/scala 下的 2.11 二进制文件的安装位置之外，2.10 jar 隐藏在/usr/share/aws/emr/emrfs/lib 中。所以我摆脱了集群所有实例中的 2.10 jar，我的工作成功完成。现在我将为此创建一个引导操作。
$ sudo find / -name "scala-library-2.10.*.jar" -exec rm -rf {} \;
仅供引用，这些是它存在的路径:
[ec2-user@ip-172-31-72-130 ~]$ sudo find / -name "scala-library-2.11.*.jar"
/home/hadoop/.versions/hbase-0.94.18/lib/scala-library-2.11.0.jar
/usr/share/doc/scala/api/jars/scala-library-2.11.1-javadoc.jar
[ec2-user@ip-172-31-72-130 ~]$ sudo find / -name "scala-library-2.10.*.jar"
/usr/share/aws/emr/emrfs/lib/scala-library-2.10.5.jar
关于scala - 在 EMR 上烫伤 : Hadoop job fails with NoSuchMethodError: scala. Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/31103525/