Java OKHTTP SocketTimeoutException 与并发多线程请求

Question

我的 Java 应用程序使用各种代理服务器从特定域收集数据。特殊申请需要办理以下手续:

• 通过特定代理加载网址
• 等待 5 秒
• 通过同一代理加载下一个网址

为了避免信息加载(由于 5 秒的暂停)永远持续下去，我总共使用了 400 个线程。每个线程都使用自己的代理服务器，即也有自己的 OKHTTP 客户端:

MyHTTPClient = new OkHttpClient.Builder()
.connectTimeout(7, TimeUnit.SECONDS)
.writeTimeout(7, TimeUnit.SECONDS)
.readTimeout(7, TimeUnit.SECONDS)
.proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(ProxyIP, ProxyPort)))
.proxyAuthenticator((Route route, Response response) - > {
    return response.request().newBuilder().header("Proxy-Authorization", Credentials.basic(ProxyUser, ProxyPass)).build();
})
.sslSocketFactory(sslSocketFactory, (X509TrustManager) trustAllCerts[0])
.hostnameVerifier((String hostname, SSLSession session) - > true)
.addNetworkInterceptor(new LoggingInterceptor())
.retryOnConnectionFailure(true)
.build();

每个线程都必须使用自己的代理，因此每个线程都有自己的 OKHTTP 客户端。总共有 400 个 OKHTTP 客户端。

每个线程现在按如下方式处理其查询:

 while (true) {
     MyAnswer = MyHTTPClient.newCall(
         new Request.Builder().url(https://www.example.com)
         .addHeader("referer", SomeReferrer)
         .addHeader("cache-control", "no-cache")
         .addHeader("pragma", "no-cache")
         .get().build())
     .execute();

     Body = MyAnswer.body().string();
     MyAnswer.body().close();

     Thread.sleep(5000);
 }

这也很有效，大约一分钟。然后，每个线程上的每个单独请求突然出现以下错误消息:

java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.waitForConnect(Native Method)
at java.base/java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:107)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.base/java.net.Socket.connect(Socket.java:608)
at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:120)
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295)
at okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.kt:261)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:201)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at okio.InputStreamSource.read(JvmOkio.kt:90)
at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:129)
... 22 more

---

到目前为止我尝试过的

我做了一些研究，发现并发连接方面可能存在某种限制。我发现，每当发生此错误时，连接池中都没有空闲连接:

MyHTTPClient.connectionPool().idleConnectionCount() //Always 0 when the Timeout-Exception occurs

所以我尝试增加每个 okhttpclient 的连接池的大小，但这似乎完全没有效果:

.connectionPool(new ConnectionPool(5000, 100, TimeUnit.MILLISECONDS))

我还尝试增加和减少 httpclient 的超时值，这实际上似乎对应用程序在没有此错误的情况下运行的时间产生了影响，但实际上并没有阻止这种情况的发生。

我还尝试增加同时连接的数量并减少连接处于“TIME_WAIT”状态的时间，如下所述:https://learn.microsoft.com/en-us/answers/questions/482793/tcpip-cuncurrent-connections.html

我还尝试向 OKHTTPClient 添加 Dispatcher，如下所示:

Dispatcher MyDispatcher = new Dispatcher(Executors.newCachedThreadPool());
MyDispatcher.setMaxRequests(9999999);
MyDispatcher.setMaxRequestsPerHost(9999999);

---

如果您对可能的原因或我还可以尝试什么有任何想法，请告诉我。如果需要，我也很乐意向您发送更多信息。

提前谢谢您!

Answer 1

首先创建一个 OkHttpClient，然后使用 OkHttpClient.newBuilder() 自定义代理。这将导致客户端共享线程池和连接池。这可能无法单独解决您的问题，但会提高效率。

  final OkHttpClient sharedHttpClient = new OkHttpClient.Builder()
    .connectTimeout(7, TimeUnit.SECONDS)
    .writeTimeout(7, TimeUnit.SECONDS)
    .readTimeout(7, TimeUnit.SECONDS)
    .sslSocketFactory(sslSocketFactory, (X509TrustManager) trustAllCerts[0])
    .hostnameVerifier((String hostname, SSLSession session) - > true)
    .addNetworkInterceptor(new LoggingInterceptor())
    .retryOnConnectionFailure(true)
    .build();

MyHTTPClient = sharedHttpClient.newBuilder()
    .proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(ProxyIP, ProxyPort)))
    .proxyAuthenticator((Route route, Response response) - > {
    return response.request().newBuilder().header("Proxy-Authorization", Credentials.basic(ProxyUser, ProxyPass)).build();
})
    .build();