java - 无法在 Elasticsearch 中对大文件进行批量索引

标签 java python elasticsearch

我正在尝试使用 Python 在 elasticsearch 中对一个大文件(~800MB)进行批量索引。但是,每次我尝试

[2016-01-08 15:06:49,354][WARN ][http.netty ] [Marvel Man] Caught exception while handling client http tra ffic, closing connection [id: 0x2d26baec, /0:0:0:0:0:0:0:1:58923 => /0:0:0:0:0:0:0:1:9200] org.jboss.netty.handler.codec.frame.TooLongFrameException: HTTP content length exceeded 104857600 bytes. at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:169) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeli ne.java:791) at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:135) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeli ne.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeli ne.java:791) at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

谁能帮我了解这里发生了什么,以及我该如何解决这个问题?

最佳答案

您必须增加默认为 100mb 的内容上传长度。 转到 config 文件夹中的 elasticsearch.yml

添加/更新-

http.max_content_length: 300M

关于java - 无法在 Elasticsearch 中对大文件进行批量索引,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/34686119/

相关文章:

python - 删除每列(和相应行)中的异常值

javascript - 我跟随文档自定义主题时使用弹性ui时遇到问题

python - 如何在python中加快Elasticsearch滚动

java - 按 "add() count"排序的集合

java - Play - Java 与 Scala 模板

java - 索尼 IP 摄像机 opencv Java

python - 如何从 Python 中的 IP 地址获取 NAPTR 记录?

java - 如何删除 eclipse rcp 应用程序中不需要的菜单贡献?

python - 尝试编译 Yolo 时,出现找不到包错误

python - 如何使用 Python API 在 Elastic Search 中批量索引