当负载超过特定大小时，Java REST API 调用失败并出现异常 "Async operation timed out"

Question

我正在一堆 Linux 服务器上设置一个自动化 Cron 作业，这需要调用 IBM WebSphere Application Server(8.5.5) 提供的一些 PUT/POST API。我用 Java 编写了自己的 REST 客户端，但奇怪的是，有时 API 执行会因以下异常而挂起。

com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]-[JAX-RS Servlet]: java.net.SocketTimeoutException: Async operation timed out
    at com.ibm.ws.tcp.channel.impl.AioTCPReadRequestContextImpl.processSyncReadRequest(AioTCPReadRequestContextImpl.java:191)
    at com.ibm.ws.tcp.channel.impl.TCPReadRequestContextImpl.read(TCPReadRequestContextImpl.java:111)
    at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.fillABuffer(HttpServiceContextImpl.java:4212)
    at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.readSingleBlock(HttpServiceContextImpl.java:3440)
    at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.readBodyBuffer(HttpServiceContextImpl.java:3546)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundServiceContextImpl.getRequestBodyBuffer(HttpInboundServiceContextImpl.java:1822)
    at com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream.bufferIsGood(WCCByteBufferInputStream.java:371)
    at com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream.read(WCCByteBufferInputStream.java:265)
    at com.ibm.ws.webcontainer.srt.http.HttpInputStream.read(HttpInputStream.java:322)
    at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:464)
    at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:506)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:234)
    at java.io.InputStreamReader.read(InputStreamReader.java:188)
    at com.ibm.json.java.JSONArray.parse(JSONArray.java:150)
    at com.ibm.websphere.jaxrs.providers.json4j.JSON4JArrayProvider.readFrom(JSON4JArrayProvider.java:44)
    at com.ibm.websphere.jaxrs.providers.json4j.JSON4JArrayProvider.readFrom(JSON4JArrayProvider.java:25)
    at org.apache.wink.server.internal.registry.ServerInjectableFactory$EntityParam.getValue(ServerInjectableFactory.java:197)
    at org.apache.wink.common.internal.registry.InjectableFactory.instantiate(InjectableFactory.java:68)
    at org.apache.wink.server.internal.handlers.CreateInvocationParametersHandler.handleRequest(CreateInvocationParametersHandler.java:44)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
    at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
    at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleSubResourceMethod(FindResourceMethodHandler.java:188)
    at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleRequest(FindResourceMethodHandler.java:110)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
    at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
    at org.apache.wink.server.internal.handlers.FindRootResourceHandler.handleRequest(FindRootResourceHandler.java:95)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
    at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
    at org.apache.wink.server.internal.handlers.HeadMethodHandler.handleRequest(HeadMethodHandler.java:53)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
    at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
    at org.apache.wink.server.internal.handlers.OptionsMethodWADLHandler.handleRequest(OptionsMethodWADLHandler.java:51)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
    at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
    at org.apache.wink.server.internal.handlers.SearchResultHandler.handleRequest(SearchResultHandler.java:33)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
    at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
    at org.apache.wink.server.internal.log.ResourceInvocation.handleRequest(ResourceInvocation.java:92)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
    at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
    at org.apache.wink.server.internal.log.Requests.handleRequest(Requests.java:76)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
    at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
    at org.apache.wink.server.handlers.AbstractHandlersChain.run(AbstractHandlersChain.java:60)
    at org.apache.wink.server.internal.RequestProcessor.handleRequestWithoutFaultBarrier(RequestProcessor.java:207)
    at org.apache.wink.server.internal.RequestProcessor.handleRequest(RequestProcessor.java:154)
    at org.apache.wink.server.internal.servlet.RestServlet.service(RestServlet.java:124)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
    at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
    at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1071)
    at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
    at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:909)
    at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
    at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:312)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:283)
    at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
    at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
    at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
    at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
    at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
    at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
    at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
    at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
    at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)
Caused by: com.ibm.io.async.AsyncTimeoutException(Async operation timed out, [Timeout, rc=0])
    at com.ibm.io.async.AbstractAsyncFuture.waitForCompletion(AbstractAsyncFuture.java:359)
    at com.ibm.io.async.AsyncFuture.getByteCount(AsyncFuture.java:218)
    at com.ibm.ws.tcp.channel.impl.AioSocketIOChannel.readAIOSync(AioSocketIOChannel.java:215)
    at com.ibm.ws.tcp.channel.impl.AioTCPReadRequestContextImpl.processSyncReadRequest(AioTCPReadRequestContextImpl.java:184)
    ... 78 more

这只发生在某些 Linux 服务器上，当我在 Windows 工作站中开发它时，它运行得非常好。所以WAS服务器中的超时设置应该不是问题。我尝试使用curl进行调试，如下所示

curl -vX PUT http://<was_server_ip>:9081/RTCWebClient/v0.1/api/testResult/syncTestSuites?testCategory=Kitchen -d @test.json --header "Content-Type: application/json"

发现当有效负载超过 1306 字节时(请参阅下面的curl 输出中的 Content-Length)，API 开始挂起并给出上述异常。

> PUT /RTCWebClient/v0.1/api/testResult/syncTestSuites?testCategory=Kitchen
HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 9.51.163.190:9081
> Accept: */*
> Content-Type: application/json
> Content-Length: 1307
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< Content-Length: 0
< Date: Wed, 24 Aug 2016 07:35:23 GMT
< HTTP/1.1 500 Internal Server Error
< Date: Wed, 24 Aug 2016 07:35:23 GMT
< X-Powered-By: Servlet/3.0
< Content-Type: text/html;charset=ISO-8859-1
< $WSEP:
< Content-Language: en-US
< Content-Length: 102
< Connection: Close
<
Error 500: javax.servlet.ServletException: java.net.SocketTimeoutException: Async operation timed out
* Closing connection #0

当有效负载小于1307字节时，API可以正常执行。

> PUT /RTCWebClient/v0.1/api/testResult/syncTestSuites?testCategory=Kitchen HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 9.51.163.190:9081
> Accept: */*
> Content-Type: application/json
> Content-Length: 1306
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< Content-Length: 0
< Date: Wed, 24 Aug 2016 07:37:51 GMT
< HTTP/1.1 200 OK
< Date: Wed, 24 Aug 2016 07:37:51 GMT
< X-Powered-By: Servlet/3.0
< Content-Type: application/json
< Access-Control-Allow-Headers: application/json
< Access-Control-Allow-Methods: GET, POST, PUT, DELETE
< Access-Control-Allow-Origin: *
< Content-Language: en-US
< Transfer-Encoding: chunked
<
* Connection #0 to host 9.51.163.190 left intact
* Closing connection #0
{"Identifier":"syncTestSuites completed","Result":{"Newly Added Test Suites":[]}}

我需要与他们的管理员核实哪些设置可能会在那些有问题的 Linux 计算机上导致此问题。但我也将其发布在这里，看看是否有人可以对这个问题提供一些深思熟虑的见解。

Answer 1

问题早就解决了，但忘记在这里更新了。

将WAS应用服务器的MTU大小从1500位降低到1320位后，问题就消失了。据网络管理员介绍，这使得服务器的数据段变小了一些，从而使某些应用程序能够更好地处理数据。

这是一篇关于最大传输单元的非常好的博客文章 https://craigocon.wordpress.com/2012/10/03/whats-mtu-and-how-do-i-change-it-in-windows-2008-r2/ 。

我对网络知识完全陌生，因此很高兴得到我们网络团队的支持。