c# - 带有 ResponseHeadersRead 的 HttpClient 在没有 Fiddler(Http/Https 调试器)的情况下在第二次 GetAsync 尝试时失败(超时)

Question

我正在尝试获取某些页面的状态代码。

问题是默认的 GetAsync 方法返回包含内容的整个页面，而我只需要标题来检查页面的状态(404,403 等)，这最终会占用内存，因为我必须检查大量的 URI .

我添加了 ResponseHeadersRead 选项来解决内存占用问题，但随后该代码开始抛出“A Task was cancelled”异常，这意味着超时。

我知道的事情:

1. 只有当我在本地 PC 上运行 fiddler(Http/Https Debugger) 时，ResponseHeadersRead 代码才有效。

2. ResponseHeadersRead 代码适用于在线编码环境，例如 dotnetfiddle。但不适用于 Windows 操作系统环境。

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;
using System.IO;
using System.Net;
using System.Security.Cryptography;


public class Program
{
    public static string[] Tags = { "first", "second" };
    public static string prefix = null;
    static HttpClient Client = new HttpClient();
    public static void Main()
    {
        System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
        Client.DefaultRequestHeaders.ConnectionClose = true;

        // limit parallel thread
        Parallel.ForEach(Tags,
        new ParallelOptions { MaxDegreeOfParallelism = Convert.ToInt32(Math.Ceiling((Environment.ProcessorCount * 0.75) * 1.0)) },
        tag =>
        {
            for (int i = 1; i < 4; i++)
            {
                switch (i)
                {
                    case 1:
                        prefix = "1";
                        break;
                    case 2:
                        prefix = "2";
                        break;
                    case 3:
                        prefix = "3";
                        break;
                }
                Console.WriteLine(tag.ToString() + " and " + i);
                HttpResponseMessage response = Client.GetAsync("https://example.com/" + prefix).Result; // this works
//                HttpResponseMessage response = Client.GetAsync("https://example.com/" + prefix,HttpCompletionOption.ResponseHeadersRead).Result; // this fails from 2nd try with one url.
                Console.WriteLine(i + " and " + (int)response.StatusCode);
                if (response.StatusCode != HttpStatusCode.NotFound)
                {

                }

            }
        });

    }
}

ResponseHeadersRead 会导致线程超时，但并非没有。

Answer 1

不要将Parallel 用于async 代码，它是为CPU 绑定(bind)而设计的。您可以同时运行所有请求，而不会浪费线程阻塞。解决这个问题的方法不是增加DefaultConnectionLimit，但是在这种情况下这将解决它。处理 ResponseHeadersRead 的正确方法是 Dispose response 即

using(HttpResponseMessage response = Client.GetAsync("https://example.com/" + prefix, HttpCompletionOption.ResponseHeadersRead).Result) {}

或读取响应的内容。

var data = response.ReadAsStringAsync().Result;

对于 ResponseHeadersRead，您需要执行此操作才能关闭连接。我鼓励您重写此代码以摆脱 Parallel 并且不要在您的 async 调用中调用 .Result。

你可以这样做:

private static async Task Go()
{
    System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
    Client.DefaultRequestHeaders.ConnectionClose = true;

    var tasks = Tags.Select(tag =>
    {
        var requests = new List<Task>();
        for (int i = 1; i < 4; i++)
        {
            switch (i)
            {
                case 1:
                    prefix = "1";
                    break;
                case 2:
                    prefix = "2";
                    break;
                case 3:
                    prefix = "3";
                    break;
            }

            requests.Add(MakeRequest(Client, prefix, tag));
        }
        return requests;
    }).SelectMany(t => t);

    await Task.WhenAll(tasks);
}

private async static Task MakeRequest(HttpClient client, string prefix, string tag)
{

    using (var response = await client.GetAsync("https://example.com/" + prefix, HttpCompletionOption.ResponseHeadersRead))
    {
        Console.WriteLine(tag + " and " + prefix);
        Console.WriteLine(prefix + " and " + (int)response.StatusCode);
    }
}