elasticsearch - 将数据从 spark 保存到 elasticsearch 时出错 - saveToEs

Question

我正在尝试将 rdd 的输出保存到 elasticsearch 中。但是当我尝试发送它时，即使包含了几个 elasticsearch-spark 库，我也会遇到错误。我是 Elasticsearch 的新手，非常感谢任何帮助。谢谢。

import org.apache.spark.{SparkConf, SparkContext}
import org.elasticsearch.spark._

object ElasticSpark {

def main(args: Array[String]) {

val logfile = "/Users/folder/Desktop/logfile.rtf";
val conf = new SparkConf().setMaster("local[1]").setAppName("RddTest");   // set master can be given any cpu cores as local[*], spark clustr, mesos,
conf.set("es.index.auto.create", "true")
val sc = new SparkContext(conf);

val logdata = sc.textFile(logfile); // number of partitions
val NumA = logdata.filter(line=>line.contains("a")).count();
val wordcount = logdata.flatMap(line=>line.split(" ")).map(word=>(word,1)).reduceByKey((a, b)=> a+ b);

println(wordcount.collect()); // doubt
wordcount.saveAsTextFile("/Users/folder/Desktop/sample") // success
wordcount.saveToEs("spark/docs")

}
}

错误

Error:(21, 15) value saveToEs is not a member of org.apache.spark.rdd.RDD[(String, Int)]
wordcount.saveToEs("spark/docs")
          ^
Error:(6, 12) object elasticsearch is not a member of package org
import org.elasticsearch.spark._
       ^

Answer 1

ES 支持不是 Spark 分发的一部分，它是 elasticsearch-hadoop 的一部分，因此您需要提供此依赖项。如果您使用 Maven，请添加到您的 pom.xml:

<dependency>
  <groupId>org.elasticsearch</groupId>
  <artifactId>elasticsearch-hadoop</artifactId>
  <version>2.2.0</version>
</dependency>

对于sbt，添加到build.sbt:

libraryDependencies += "org.elasticsearch" % "elasticsearch-hadoop" % "2.2.0" % "compile"
resolvers ++= Seq("clojars" at "https://clojars.org/repo",
                  "conjars" at "http://conjars.org/repo",
                  "plugins" at "http://repo.spring.io/plugins-release",
                  "sonatype" at "http://oss.sonatype.org/content/groups/public/")