c# - 是否可以使用 ConcurrentExclusiveSchedulerPair 优化这段代码？

Question

The problem is currently it runs pretty fine with limited number of reports (around 10 000), but on environments with much more reports it fails with Big amount of simultaneous requests for server causing heavy load on the server:

private void PerformSearch(SearchReportModel item)
{
    var tasks = new List<Task>();

    foreach (var term in item.SearchTerms)
    {
        var model = GetBaseQueryModel(item.Site);
        tasks.Add(Task.Factory.StartNew(() => CheckSearchTerm(model, term, item.Site, item.Language)));
    }
    try
    {
        Task.WaitAll(tasks.ToArray());
    }            
    catch (Exception ex)
    {
        Log4NetLogger.LogError(ex, ex.ToString());
        throw;
    }
}

Answer 1

假设 CheckSearchTerm 执行有问题的 API 调用，我建议仅使用 SemaphoreSlim限制负载:

Represents a lightweight alternative to Semaphore that limits the number of threads that can access a resource or pool of resources concurrently.

var tasks = new List<Task>();
var maximumRequests = 100; // maximum simultaneous invocations 
var limiter = new SemaphoreSlim(maximumRequests); // possibly make a global one via static variable or move inside of CheckSearchTerm
foreach (var term in item.SearchTerms)
{
    var model = GetBaseQueryModel(item.Site);
    tasks.Add(Task.Run(async () =>
    {
        await limiter.WaitAsync();
        try
        {
            CheckSearchTerm(model, term, item.Site, item.Language);
        }
        finally
        {
            limiter.Release();
        }
        
    }));
}

另一个选项是使用 Parallel.ForEach/PLINQ将 ParallelOptions.MaxDegreeOfParallelism/WithDegreeOfParallelism 设置为所需值。