c# - 多个同时的 WebClient http 调用完成速度很慢

Question

我在 IIS 中托管的 Web 服务中遇到启动问题。该服务需要通过 http 在线获取外部资源，以便在服务请求时使用。为了简化这一点，这些资源在后台线程中收集并缓存。而且由于我不希望对服务的初始请求失败，所以这些资源在启动期间会立即收集。它们是使用 System.Net.WebClient 类下载的，所发生的事情是创建了一堆线程，它们都有一个 WebClient 对象，用于下载资源。

这很奇怪。就好像多个请求以某种方式互相阻塞一样，因为发生的情况是所有请求都需要很长时间才能完成。当所有这些请求都在运行时，我可以调用本地托管的网站，该网站由一个小的“hello world”http 页面组成，这将花费几秒钟和几秒钟(如果在后台线程之前调用，我当前的设置需要 0.1 秒)如果在启动后调用则为 30)。大约一分钟后一切恢复正常，而不是无法访问任何资源，它可以在不到一秒的时间内突然下载大部分资源。

实验表明，这不是并发线程或连接限制的问题(一分钟内根本没有任何成功的调用 - 如果存在某种连接限制，您会期望一些调用成功) .另一个有趣的实验是当我改为尝试使用纯套接字连接到小型本地站点时 - 这工作完美无缺，所以这似乎是 WebClient 类的一个问题。

用于建立连接的类如下所示。对 TimeoutRequestHandler 的调用非常简单 - 您使用 uri 和 CustomWebClient 创建它，然后调用处理请求。

    private class CustomWebClient : WebClient {

        public bool KeepAlive { get; set; }

        public X509Certificate ClientCertificate { get; set; }

        protected override WebRequest GetWebRequest(Uri address) {

            WebRequest answer = base.GetWebRequest(address);

            HttpWebRequest httpReq = answer as HttpWebRequest;
            if (httpReq != null) {
                httpReq.KeepAlive = KeepAlive;

                if (ClientCertificate != null) {
                    httpReq.ClientCertificates.Add(ClientCertificate);
                }
            }
            return answer;
        }
    }

    private class TimeoutRequestHandler {

        private readonly Uri address;
        private readonly WebClient client;
        private readonly byte[] requestData;
        private readonly TimeSpan timeout;

        private readonly object sync;
        private ManualResetEvent requestDoneSignal;
        private AsyncCompletedEventArgs completedInfo;

        public TimeoutRequestHandler(Uri address, WebClient client, byte[] requestData, TimeSpan timeout) {

            this.address = address;
            this.client = client;
            this.requestData = requestData;
            this.timeout = timeout;

            sync = new object();
            requestDoneSignal = new ManualResetEvent(false);
            client.UploadDataCompleted += OnRequestCompleted;
            client.DownloadDataCompleted += OnRequestCompleted;
        }

        public byte[] ProcessRequest() {

            bool shouldCancel = false;
            try {
                if (requestData != null) {
                    client.UploadDataAsync(address, requestData); // Uses POST for HTTP.
                } else {
                    client.DownloadDataAsync(address); // Uses GET for HTTP
                }
                if (!requestDoneSignal.WaitOne(timeout)) {
                    shouldCancel = true;
                    throw new WebException("The operation has timed out");
                }
                if (completedInfo.Cancelled) {
                    throw new WebException("The operation has been cancelled");
                }
                if (completedInfo.Error != null) {
                    throw completedInfo.Error;
                }
                return GetResponseData(completedInfo);
            } finally {
                AllDone(shouldCancel);
            }
        }

        private byte[] GetResponseData(AsyncCompletedEventArgs e) {

            return e is UploadDataCompletedEventArgs ?
                ((UploadDataCompletedEventArgs)e).Result :
                ((DownloadDataCompletedEventArgs)e).Result;
        }

        private void OnRequestCompleted(object sender, AsyncCompletedEventArgs e) {

            lock (sync) {
                if (requestDoneSignal != null) {
                    completedInfo = e;
                    requestDoneSignal.Set();
                }
            }
        }

        private void AllDone(bool shouldCancel) {

            lock (sync) {
                requestDoneSignal.Close();
                requestDoneSignal = null;
                client.UploadDataCompleted -= OnRequestCompleted;
                client.DownloadDataCompleted -= OnRequestCompleted;
                if (shouldCancel) {
                    client.CancelAsync();
                }
            }
        }
    }
}

Answer 1

不确定它是否相关，但我已经看到解析代理服务器可能非常缓慢并增加大量开销的问题。我最近在一个项目中添加了以下行以提高 WebClient 性能:

ServicePointManager.DefaultConnectionLimit = 256;
WebRequest.DefaultWebProxy = null;

ServicePointManager.DefaultConnectionLimit Property :

The DefaultConnectionLimit property sets the default maximum number of concurrent connections that the ServicePointManager object assigns to the ConnectionLimit property when creating ServicePoint objects.

ServicePointManager.DefaultConnectionLimit 的文档并不明确，因为它指出:

The default value is Int32.MaxValue.

然后继续说:

When used in the server environment (ASP.NET) DefaultConnectionLimit defaults to higher number of connections, which is 10.

我刚刚在 Visual Studio 2015 中查看了 .NET 4.5 WPF 应用程序中的值，它默认为 2:

WebRequest.DefaultWebProxy Property :

The DefaultWebProxy property gets or sets the global proxy. The DefaultWebProxy property determines the default proxy that all WebRequest instances use if the request supports proxies and no proxy is set explicitly using the Proxy property. Proxies are currently supported by FtpWebRequest and HttpWebRequest.

The DefaultWebProxy property reads proxy settings from the app.config file. If there is no config file, the current user's Internet Explorer (IE) proxy settings are used.

If the DefaultWebProxy property is set to null, all subsequent instances of the WebRequest class created by the Create or CreateDefault methods do not have a proxy.

以下配置参数也可用:

<system.net>
  <connectionManagement>
    <add address="*" maxconnection="256"/>
  </connectionManagement>
  <defaultProxy enabled="false" />
</system.net>

<connectionManagement> Element (Network Settings)

<defaultProxy> Element (Network Settings)

显然，如果您有代理，则在 defaultProxy 上设置 enabled="false" 将不起作用。在那种情况下，我会在配置中指定代理详细信息，这样它就不必检查 IE。