c# - ASP.NET Core 禁用响应缓冲

Question

我正在尝试将动态构建的大型 JSON 文件流式传输到客户端(可能是 500 MB+)。由于各种原因，我试图禁用响应缓冲，但主要是为了提高内存效率。
我试过直接写信给 HttpContext.Response.BodyWriter但响应似乎在写入输出之前缓冲在内存中。此方法的返回类型为 Task .

HttpContext.Response.ContentType = "application/json";
HttpContext.Response.ContentLength = null;
await HttpContext.Response.StartAsync(cancellationToken);
var bodyStream = HttpContext.Response.BodyWriter.AsStream(true);
await bodyStream.WriteAsync(Encoding.UTF8.GetBytes("["), cancellationToken);
await foreach (var item in cursor.WithCancellation(cancellationToken)
    .ConfigureAwait(false))
{
    await bodyStream.WriteAsync(JsonSerializer.SerializeToUtf8Bytes(item, DefaultSettings.JsonSerializerOptions), cancellationToken);
    await bodyStream.WriteAsync(Encoding.UTF8.GetBytes(","), cancellationToken);
    
    await bodyStream.FlushAsync(cancellationToken);
    await Task.Delay(100,cancellationToken);
}
await bodyStream.WriteAsync(Encoding.UTF8.GetBytes("]"), cancellationToken);
bodyStream.Close();
await HttpContext.Response.CompleteAsync().ConfigureAwait(false);

注意:我意识到这段代码很hacky，试图让它工作，然后清理它
我正在使用 Task.Delay因为我没有完整的生产数据，所以在本地测试时验证响应没有被缓冲。我也试过IAsyncEnumerable和 yield return ，但这失败了，因为响应太大，以至于 Kestrel 认为可枚举是无限的。
我试过了

设置 KestrelServerLimits.MaxResponseBufferSize到一个很小的数字，甚至是 0；

写作 HttpContext.Response.WriteAsync

写作 HttpContext.Response.BodyWriter.AsStream()

用管道作家模式写作和HttpContext.Response.BodyWriter

删除所有中间件

删除对 IApplicationBuilder.UseResponseCompression 的调用

更新

在设置 ContentType 之前尝试禁用响应缓冲(所以在任何写入响应之前)没有影响

var responseBufferingFeature = context.Features.Get<IHttpResponseBodyFeature>();
responseBufferingFeature?.DisableBuffering();

更新的示例代码
这很简单地重现了这个问题。客户端直到 response.CompleteAsync() 才收到任何数据叫做。

[HttpGet]
[Route("stream")]
public async Task<EmptyResult> FileStream(CancellationToken cancellationToken)
{
    var response = DisableResponseBuffering(HttpContext);
    HttpContext.Response.Headers.Add("Content-Type", "application/gzip");
    HttpContext.Response.Headers.Add("Content-Disposition", $"attachment; filename=\"player-data.csv.gz\"");
    await response.StartAsync().ConfigureAwait(false);
    var memory = response.Writer.GetMemory(1024*1024*10);
    response.Writer.Advance(1024*1024*10);
    await response.Writer.FlushAsync(cancellationToken).ConfigureAwait(false);
    await Task.Delay(5000).ConfigureAwait(false);
    var str2 = Encoding.UTF8.GetBytes("Bar!\r\n");
    memory = response.Writer.GetMemory(str2.Length);
    str2.CopyTo(memory);
    response.Writer.Advance(str2.Length);
    await response.CompleteAsync().ConfigureAwait(false);
    return new EmptyResult();
}

private IHttpResponseBodyFeature DisableResponseBuffering(HttpContext context)
{
    var responseBufferingFeature = context.Features.Get<IHttpResponseBodyFeature>();
    responseBufferingFeature?.DisableBuffering();
    return responseBufferingFeature;
}

Answer 1

尝试禁用响应 future 的缓冲:

HttpContext.Features.Get<IHttpResponseBodyFeature>().DisableBuffering()
//As mentioned in documentation, to take effect, call it before any writes

并使用 BodyWriter在 Utf8JsonWriter提高效率:

 var pipe = context.HttpContext.Response.BodyWriter;
 await pipe.WriteAsync(startArray);
 using (var writer = new Utf8JsonWriter(pipe,
            new JsonWriterOptions
            {
                Indented = option.WriteIndented,
                Encoder = option.Encoder,
                SkipValidation = true
            }))
 {
      var dotSet = false;
      foreach (var item in enumerable)
      {
           if (dotSet)
               await pipe.WriteAsync(dot);
           JsonSerializer.Serialize(writer, item, itemType, option);
           await pipe.FlushAsync();
           writer.Reset();
           dotSet = true;
      }
 }
 await pipe.WriteAsync(endArray);

就我而言，它给出了结果:在第一次请求后，总内存分配比 newcoreapp2.2 增加了 80% 以上，但没有更多的内存泄漏。