python - 由于连接错误，Spark保存到elasticsearch失败

Question

我正在尝试对Elasticsearch中具有150个字段的77M文档的数据进行索引。
我们没有太多的计算/内存资源，因此我们的集群是3个节点(48GB RAM / 24 CPU和6TB存储空间)

我正在从另一个虚拟网络中的另一个Spark集群发送数据，但两个网络已配对，并且我可以从Spark集群节点ping所有els节点。

我面临的问题是:在索引了一定数量的文档(大约8M)时，spark无法连接到els，并引发以下错误:

Job aborted due to stage failure: Task 173 in stage 9.0 failed 4 times, most recent failure: Lost task 173.3 in stage 9.0 (TID 17160, wn21-swspar.of12wietsveu3a3voc5bflf1pa.ax.internal.cloudapp.net, executor 3): org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[10.0.0.12:9200]] 
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:149)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:466)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:450)
at org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.java:186)
at org.elasticsearch.hadoop.rest.RestRepository.tryFlush(RestRepository.java:248)
at org.elasticsearch.hadoop.rest.RestRepository.flush(RestRepository.java:270)
at org.elasticsearch.hadoop.rest.RestRepository.doWriteToIndex(RestRepository.java:210)
at org.elasticsearch.hadoop.rest.RestRepository.writeToIndex(RestRepository.java:187)
at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:67)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:101)
at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:101)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

我不知道是什么原因造成的。集群大小(RAM / CPU)是否足够，或者对于具有大量数据的索引是否有特殊的配置？
我确定这不是网络问题。
ELS版本:6.2.4

Answer 1

我相信Spark会尝试发出太多您的ES集群无法处理的请求。
检查此链接https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html可以进一步调整如何将数据发送到ES，请检查以下内容:

es.batch.size.bytes

es.batch.size.entries

es.batch.write.retry.count

es.http.retries

es.http.timeout

并且也请检查您要如何使用es.nodes.client.only等方式路由请求...