elasticsearch - Reindex调用中 “include”参数中指定的字段数是否有限制

Question

这是我在执行Reindex调用时遇到的错误消息

[2020-03-13T16:08:39,042][WARN ][o.e.i.r.TransportReindexAction] [ExportDatabase] giving up on search because it failed with a non-retryable exception org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:288) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:128) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:249) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:222) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:176) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:222) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:527) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1095) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1188) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1172) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.SearchTransportService$6$1.onFailure(SearchTransportService.java:385) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:341) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:329) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1019) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_212] at java.lang.Thread.run(Unknown Source) [?:1.8.0_212] Caused by: org.elasticsearch.ElasticsearchException$1: Determinizing automaton with 33501 states and 34977 transitions would result in more than 10000 states. at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:658) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:126) ~[elasticsearch-6.3.0.jar:6.3.0] ... 26 more Caused by: org.apache.lucene.util.automaton.TooComplexToDeterminizeException: Determinizing automaton with 33501 states and 34977 transitions would result in more than 10000 states. at org.apache.lucene.util.automaton.Operations.determinize(Operations.java:746) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24] at org.apache.lucene.util.automaton.RunAutomaton.(RunAutomaton.java:69) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24] at org.apache.lucene.util.automaton.CharacterRunAutomaton.(CharacterRunAutomaton.java:39) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24] at org.apache.lucene.util.automaton.CharacterRunAutomaton.(CharacterRunAutomaton.java:28) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24] at org.elasticsearch.common.xcontent.support.XContentMapValues.filter(XContentMapValues.java:174) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.fetch.subphase.FetchSourceContext.getFilter(FetchSourceContext.java:234) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.lookup.SourceLookup.filter(SourceLookup.java:132) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.fetch.subphase.FetchSourceSubPhase.hitExecute(FetchSourceSubPhase.java:56) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:162) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:393) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:368) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:333) ~[elasticsearch-6.3.0.jar:6.3.0] ... 9 more

因此，根据我的分析，这是我正在执行的查询:

POST _reindex
{
"source":{
"index":"index_source",
"size":1000,
"query":
{
"terms":{
"ID":["7078"],
"boost":1.0
}
},
"_source":
{
"includes":here we have 1216 fields]
}
},
"dest":
{
"index":"index_dest"
}
}

在这里，如果我删除包含参数中的1216个字段，或者将其减少到500个以下，那么它将起作用

因此，在这方面需要帮助，是否有其他方法我们只能为这些特定字段重新编制索引，或者我做错了什么。

Answer 1

从 flex 搜索文档Mapping

1000是索引中允许的最大字段数。
要增加限制，您可以更新“index.mapping.total_fields.limit”

警告

The limit is in place to prevent mappings and searches from becoming too large. Higher values can lead to performance degradations and memory issues, especially in clusters with a high load or few resources. If you increase this setting, we recommend you also increase the indices.query.bool.max_clause_count setting, which limits the maximum number of boolean clauses in a query.