我正在尝试在 scala 中集成 Spark 和 Elasticsearch,如 Elasticsearch Guide 中所述
我在编译时遇到依赖问题:
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: cascading#ing-local;2.5.6: not found
[error] unresolved dependency: clj-time#clj-time;0.4.1: not found
[error] unresolved dependency: compojure#compojure;1.1.3: not found
[error] unresolved dependency: hiccup#hiccup;0.3.6: not found
[error] unresolved dependency: ring#ring-devel;0.3.11: not found
[error] unresolved dependency: ring#ring-jetty-adapter;0.3.11: not found
[error] unresolved dependency: com.twitter#carbonite;1.4.0: not found
[error] unresolved dependency: cascading#cascading-hadoop;2.5.6: not found
[error] Total time: 86 s, completed 19 nov. 2014 08:42:58
我的 build.sbt 文件是这样的
name := "twitter-sparkstreaming-elasticsearch"
version := "0.0.1"
scalaVersion := "2.10.4"
// additional libraries
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.1.0",
"org.apache.spark" %% "spark-streaming" % "1.1.0",
"org.apache.spark" %% "spark-streaming-twitter" % "1.1.0",
"org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0"
)
帮助? 谢谢。
最佳答案
Cascading 及其依赖项在 Maven central 中不可用,但在它们自己的 repo 中可用(es-hadoop 无法通过其 pom 指定)。
我使用 elasticsearch-spark_2.10 解决了这个问题
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/master/install.html
关于scala - Spark 流式 Elasticsearch 依赖项,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/27015172/