c# - 管理多个 WebRequest 的更好方法

Question

我有一个组件在单独的线程中处理多个 Web 请求。每个 WebRequest 处理都是同步的。

public class WebRequestProcessor:System.ComponentModel.Component
{
    List<Worker> tlist = new List<Worker>();
    public void Start()
    {
        foreach(string url in urlList){
            // Create the thread object. This does not start the thread.
            Worker workerObject = new Worker();
            Thread workerThread = new Thread(workerObject.DoWork);

            // Start the worker thread.
            workerThread.Start(url);
            tlist.Add(workerThread);
        }
    }
}

public class Worker
{
    // This method will be called when the thread is started.
    public void DoWork(string url)
    {
        // prepare the web page we will be asking for
        HttpWebRequest  request  = (HttpWebRequest) 
            WebRequest.Create(url);

        // execute the request
        HttpWebResponse response = (HttpWebResponse)
            request.GetResponse();

        // we will read data via the response stream
        Stream resStream = response.GetResponseStream();

        // process stream
    }
}

现在我必须找到取消所有请求的最佳方式。

一种方法是将每个同步WebRequest 转换为async 并使用WebRequest.Abort 取消处理。

另一种方法是释放线程指针并允许所有线程使用 GC 死亡。

Answer 1

如果要下载 1000 个文件，一次启动 1000 个线程当然不是最佳选择。与一次只下载几个文件相比，它不仅不会给您带来任何加速，而且还需要至少 1 GB 的虚拟内存。创建线程是昂贵的，尽量避免在循环中这样做。

您应该改为使用 Parallel.ForEach() 以及请求和响应操作的异步版本。例如像这样(WPF 代码):

private void Start_Click(object sender, RoutedEventArgs e)
{
    m_tokenSource = new CancellationTokenSource();
    var urls = …;
    Task.Factory.StartNew(() => Start(urls, m_tokenSource.Token), m_tokenSource.Token);
}

private void Cancel_Click(object sender, RoutedEventArgs e)
{
    m_tokenSource.Cancel();
}

void Start(IEnumerable<string> urlList, CancellationToken token)
{
    Parallel.ForEach(urlList, new ParallelOptions { CancellationToken = token },
                     url => DownloadOne(url, token));

}

void DownloadOne(string url, CancellationToken token)
{
    ReportStart(url);

    try
    {
        var request = WebRequest.Create(url);

        var asyncResult = request.BeginGetResponse(null, null);

        WaitHandle.WaitAny(new[] { asyncResult.AsyncWaitHandle, token.WaitHandle });

        if (token.IsCancellationRequested)
        {
            request.Abort();
            return;
        }

        var response = request.EndGetResponse(asyncResult);

        using (var stream = response.GetResponseStream())
        {
            byte[] bytes = new byte[4096];

            while (true)
            {
                asyncResult = stream.BeginRead(bytes, 0, bytes.Length, null, null);

                WaitHandle.WaitAny(new[] { asyncResult.AsyncWaitHandle,
                                           token.WaitHandle });

                if (token.IsCancellationRequested)
                    break;

                var read = stream.EndRead(asyncResult);

                if (read == 0)
                    break;

                // do something with the downloaded bytes
            }
        }

        response.Close();
    }
    finally
    {
        ReportFinish(url);
    }
}

这样，当您取消操作时，所有下载都将被取消并且不会开始新的下载。此外，您可能想要设置 ParallelOptions 的 MaxDegreeOfParallelism，这样您就不会一次下载太多。

我不确定您要对正在下载的文件做什么，因此使用 StreamReader 可能是更好的选择。