java - 突然出现此错误 java.lang.IllegalStateException : Request cannot be executed; I/O reactor status: STOPPED

Question

我正在使用httpcore-nio-4.4.5.jar。我正在使用elasticsearch RestHighLevelClient 与我们的elasticsearch 服务器交互。这一切都工作正常，除了有时我们会突然出现 I/O react 器停止错误。

ES 方面一切似乎都很好。没有奇怪的行为。

这就是我初始化 ES 客户端的方式。

public synchronized RestHighLevelClient getHighLevelClient() throws ManagerException {
        if (highLevelClient != null) {
            return highLevelClient;
        }

        Map<String, Integer> map = getEsServers(esAddresses);

        HttpHost[] hosts = new HttpHost[map.size()];

        int i = 0;

        for (Map.Entry<String, Integer> entry : map.entrySet()) {
            hosts[i++] = new HttpHost(entry.getKey(), entry.getValue(), "http");
            LOGGER.info(entry.getKey() + " " + entry.getValue());
        }

        RestClientBuilder restClientBuilder = RestClient.builder(hosts);
        highLevelClient = customizeHttpClient(restClientBuilder);
        return highLevelClient;
    }

public RestHighLevelClient customizeHttpClient(RestClientBuilder restClientBuilder) {
        Header[] defaultHeaders = new Header[2];
        defaultHeaders[0] = new BasicHeader("Authorization", "Basic YTph");
        defaultHeaders[1] = new BasicHeader("Accept", "application/json");

        restClientBuilder.setDefaultHeaders(defaultHeaders);

        restClientBuilder.setMaxRetryTimeoutMillis(MAX_RETRY_TIMEOUT_MILLIS);

        restClientBuilder.setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder
                .setConnectTimeout(CONNECT_TIMEOUT_MILLIS)
                .setSocketTimeout(SOCKET_TIMEOUT_MILLIS)
                .setConnectionRequestTimeout(CONNECTION_REQUEST_TIMEOUT_MILLIS));

        restClientBuilder.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder
                .setMaxConnPerRoute(MAX_CONN_PER_ROUTE)
                .setMaxConnTotal(MAX_CONN_TOTAL));

        return new RestHighLevelClient(restClientBuilder);
    }

所以基本上首先我得到以下堆栈跟踪

java.lang.IllegalStateException: I/O reactor has been shut down
        at org.apache.http.util.Asserts.check(Asserts.java:34) 
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.connect(DefaultConnectingIOReactor.java:224) 
        at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:434) 
        at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:276) 
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:266) 
        at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:363) 
        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:125) 
        at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) 
        at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:346) 
        at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:328) 
        at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:271) 
        at org.elasticsearch.client.RestHighLevelClient.performRequestAsync(RestHighLevelClient.java:537) 
        at org.elasticsearch.client.RestHighLevelClient.performRequestAsyncAndParseEntity(RestHighLevelClient.java:515) 
        at org.elasticsearch.client.RestHighLevelClient.searchAsync(RestHighLevelClient.java:400) 

之后，没有超时，只是不断出现以下异常，直到我重新启动服务器。

java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
    at org.apache.http.util.Asserts.check(Asserts.java:46)  
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)  
    at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)  
    at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:346)  
    at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:328)  
    at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:271)  
    at org.elasticsearch.client.RestHighLevelClient.performRequestAsync(RestHighLevelClient.java:537)  
    at org.elasticsearch.client.RestHighLevelClient.performRequestAsyncAndParseEntity(RestHighLevelClient.java:515)  
    at org.elasticsearch.client.RestHighLevelClient.searchAsync(RestHighLevelClient.java:400)  ```

Answer 1

我遇到了同样的问题，结果发现我没有关闭 HighLevelRestClient。

请参阅 HenningAndersen 的回答 https://discuss.elastic.co/t/request-cannot-be-executed-i-o-reactor-status-stopped/195438/4 :

This sounds similar to https://github.com/elastic/elasticsearch/issues/45115 141. The problem in that issue is caused by throwing an exception in an onFailure method in the client application code. This causes the connection to be closed and subsequent requests will fail with the error reported here.

I believe this also affects the high level client if using any of the methods reporting response/failure back through an ActionListener (I think all of them have suffix Async in their method names).

The workaround is to ensure that no exceptions are thrown out of your onFailure methods.