io - Erlang消费者队列

Question

我有一个问题，我想将离散的数据 block 从磁盘拉入队列，并将它们出列到另一个进程中。该数据随机位于磁盘上，因此不会从顺序读取中获得很大好处。数据量很大，所以我无法一次加载全部数据，一次拉入一个 block 也效率不高。

我希望消费者能够以自己的速度运行，但要为其准备好一个健康的数据队列，这样我在处理 block 时就不会一直等待磁盘读取。

有没有既定的方法可以做到这一点？即工作框架或安全阀？实现这一点感觉就像重新发明轮子，因为对磁盘数据操作缓慢的消费者是一个常见问题。

关于如何最好地以 Erlang 方式解决这个问题，有什么建议吗？

Answer 1

您可以在file:open/2上使用{read_ahead, Bytes}选项:

{read_ahead, Size}

This option activates read data buffering. If read/2 calls are for significantly less than Size bytes, read operations towards the operating system are still performed for blocks of Size bytes. The extra data is buffered and returned in subsequent read/2 calls, giving a performance gain since the number of operating system calls is reduced.

The read_ahead buffer is also highly utilized by the read_line/1 function in raw mode, why this option is recommended (for performance reasons) when accessing raw files using that function.

If read/2 calls are for sizes not significantly less than, or even greater than Size bytes, no performance gain can be expected.

您对提到的使用大小一直含糊其辞，但似乎调整该缓冲区大小应该是实现您所需内容的一个不错的开始。