node.js - NodeJs 流管道是对称的吗？

Question

我正在构建一个将文件从端点 A 传输到端点 B 的服务器。

我想知道 NodeJs 流管道是否对称？ 如果我执行以下操作:request.get(A).pipe(request.put(B));，它的上传速度是否与下载速度一样快？

我问这个问题是因为我的服务器有非对称连接(下载速度比上传速度快)，并且我尽力避免内存消耗。

Answer 1

根据 Node 的documentation on stream#pipe Pipe 会将读取流切换到流动模式 - 仅当写入流完成消耗先前的数据包时才会读取。

readable.pipe() method attaches a Writable stream to the readable, causing it to switch automatically into flowing mode and push all of its data to the attached Writable. The flow of data will be automatically managed so that the destination Writable stream is not overwhelmed by a faster Readable stream.

因此，由于发送/下载速度不同，您的传输可能是不对称的 - 差异可能会缓冲在 Node 的内存中 - Buffering of streams

Buffering#

Both Writable and Readable streams will store data in an internal buffer that can be retrieved using writable._writableState.getBuffer() or readable._readableState.buffer, respectively.

The amount of data potentially buffered depends on the highWaterMark option passed into the streams constructor. For normal streams, the highWaterMark option specifies a total number of bytes. For streams operating in object mode, the highWaterMark specifies a total number of objects.

Data is buffered in Readable streams when the implementation calls stream.push(chunk). If the consumer of the Stream does not call stream.read(), the data will sit in the internal queue until it is consumed.

Once the total size of the internal read buffer reaches the threshold specified by highWaterMark, the stream will temporarily stop reading data from the underlying resource until the data currently buffered can be consumed (that is, the stream will stop calling the internal readable._read() method that is used to fill the read buffer).

Data is buffered in Writable streams when the writable.write(chunk) method is called repeatedly. While the total size of the internal write buffer is below the threshold set by highWaterMark, calls to writable.write() will return true. Once the the size of the internal buffer reaches or exceeds the highWaterMark, false will be returned.

A key goal of the stream API, and in particular the stream.pipe() method, is to limit the buffering of data to acceptable levels such that sources and destinations of differing speeds will not overwhelm the available memory.

Because Duplex and Transform streams are both Readable and Writable, each maintain two separate internal buffers used for reading and writing, allowing each side to operate independently of the other while maintaining an appropriate and efficient flow of data. For example, net.Socket instances are Duplex streams whose Readable side allows consumption of data received from the socket and whose Writable side allows writing data to the socket. Because data may be written to the socket at a faster or slower rate than data is received, it is important each side operate (and buffer) independently of the other.

我建议您查看this question这里对该主题进行进一步阐述。

如果您运行以下示例

const http = require('http');

http.request({method:'GET', host:'somehost.com', path: '/cat-picture.jpg'}, (response)=>{
  console.log(response);
}).end()

您可以探索底层套接字 - 在我的系统上，它们都具有 highWaterMark : 16384 属性。因此，如果我理解文档和上述问题，在您的情况下，大约 16KB 可能会在 Node.js 级别上更快的 GET 套接字中缓冲 - 下面发生的情况可能高度依赖于您的系统/网络配置。