java - DefaultClientConnection 自动关闭

Question

我是 HttpClient 的新手。我正在使用 DefaultHttpClient (据说是线程安全的)。 在我的应用程序中，我创建了两个线程，计划每 10 分钟同时执行一次。 有时我发现DefaultClientConnection会自动关闭。

1. 可能的原因是什么？
2. 虽然 DefaultHttpClient 是线程安全的，但在这种情况下我是否需要使用 PoolingClientConnectionManager ？

Answer 1

您应该使用 PoolingClientConnectionManager。 另外，您必须使用 IdleConnectionMonitorThread 来监视空闲连接。

来 self 来源的一些代码:

private final PoolingClientConnectionManager connectionManager;
private final IdleConnectionMonitorThread connectionMonitorThread = null;
private final DefaultHttpClient httpclient;

初始化:

    final HttpParams params = new BasicHttpParams();
    final HttpProtocolParamBean paramsBean = new HttpProtocolParamBean(params);
    paramsBean.setVersion(HttpVersion.HTTP_1_1);
    paramsBean.setContentCharset("UTF-8");
    paramsBean.setUseExpectContinue(false);
    params.setBooleanParameter(HttpConnectionParams.STALE_CONNECTION_CHECK, false);
    params.setIntParameter("http.socket.timeout", 20000);
    params.setIntParameter("http.connection.timeout", 30000);
    params.setBooleanParameter("http.protocol.handle-redirects", true);
    params.setBooleanParameter(HttpConnectionParams.TCP_NODELAY, true);

    params.setIntParameter(HttpConnectionParams.SOCKET_BUFFER_SIZE, 32 * 1024);

    params.setParameter("http.protocol.cookie-policy", CookiePolicy.BROWSER_COMPATIBILITY);
    // params.setParameter("http.useragent", "Crawler Airupt(http://www.airupt.com/)");
    params.setParameter("http.useragent",
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.79 Safari/537.1");
    params.setParameter("http.language.Accept-Language", "en-us");
    params.setParameter("http.protocol.content-charset", "UTF-8");
    params.setParameter("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
    params.setParameter("Cache-Control", "max-age=0");

    final SSLSocketFactory socketFactory = SSLSocketFactory.getSocketFactory();
    final SchemeRegistry schemeRegistry = new SchemeRegistry();
    schemeRegistry.register(new Scheme("http", 80, PlainSocketFactory.getSocketFactory()));
    schemeRegistry.register(new Scheme("https", 443, socketFactory/* SSLSocketFactory.getSocketFactory() */));

    connectionManager = new PoolingClientConnectionManager(schemeRegistry);
    connectionManager.setDefaultMaxPerRoute(500000);
    connectionManager.setMaxTotal(2000000);

    httpclient = new DefaultHttpClient(connectionManager, params);
    httpclient.setKeepAliveStrategy(new ConnectionKeepAliveStrategy() {
        @Override
        public long getKeepAliveDuration(final HttpResponse response, final HttpContext context) {
            final HeaderElementIterator it = new BasicHeaderElementIterator(response
                    .headerIterator(HTTP.CONN_KEEP_ALIVE));
            while (it.hasNext()) {
                final HeaderElement he = it.nextElement();
                final String param = he.getName();
                final String value = he.getValue();
                if (value != null && param.equalsIgnoreCase("timeout")) {
                    try {
                        return Long.parseLong(value) * 1000;
                    } catch (final NumberFormatException ignore) {
                    }
                }
            }
            return 30 * 1000;
        }
    });

    httpclient.setRedirectStrategy(new DefaultRedirectStrategy());

    httpclient.addRequestInterceptor(new HttpRequestInterceptor() {

        @Override
        public void process(final HttpRequest request, final HttpContext context) throws HttpException, IOException {
            if (!request.containsHeader("Accept-Encoding")) {
                request.addHeader("Accept-Encoding", "gzip");
            }
        }

    });

    httpclient.addResponseInterceptor(new HttpResponseInterceptor() {

        @Override
        public void process(final HttpResponse response, final HttpContext context) throws HttpException, IOException {
            response.setEntity(new BufferedHttpEntity(response.getEntity()));
            final HttpEntity entity = response.getEntity();
            final Header ceheader = entity.getContentEncoding();
            if (ceheader != null) {
                final HeaderElement[] codecs = ceheader.getElements();
                for (int i = 0; i < codecs.length; i++) {
                    if (codecs[i].getName().equalsIgnoreCase("gzip")) {
                        response.setEntity(new GzipDecompressingEntity(response.getEntity()));
                        return;
                    }
                }
            }
        }

    });

    startConnectionMonitorThread();

几种添加方法:

    private synchronized void startConnectionMonitorThread() {
    if (connectionMonitorThread == null) {
        connectionMonitorThread = new IdleConnectionMonitorThread(connectionManager);
    }
    connectionMonitorThread.start();
}

private synchronized void stopConnectionMonitorThread() {
    if (connectionMonitorThread != null) {
        connectionMonitorThread.shutdown();
        connectionManager.shutdown();
    }
}
    public void shutdown() {
    stopConnectionMonitorThread();
    final ClientConnectionManager cm = httpclient.getConnectionManager();
    if (cm != null) {
        httpclient.getConnectionManager().shutdown();
    }
}

使用:

final HttpGet httpGet = new HttpGet(url);
final HttpResponse response = httpclient.execute(httpGet);
final StatusLine statusLine = response.getStatusLine();
final int responseCode = statusLine.getStatusCode();
if (responseCode >= 300) {
    logger.error(" {}. Received statusCode {}", url, responseCode);
    httpGet.abort();
    //throw some exception;
}
final HttpEntity entity = response.getEntity();
if (entity == null) {
    //throw some exception or ignore;
}
                responseBody = EntityUtils.toString(entity);

此代码/参数针对爬虫进行了优化。快速接收大量页面。使用 gzip(如果可能)和 https(如果需要)而不使用 cookie。对于添加 cookie，您需要添加 cookieStore，例如 httpclient.setCookieStore();