java - 使用 Java MappedByteBuffer 进行并发读取

Question

我正在尝试使用 MappedByteBuffer 来允许多个线程在具有以下约束的情况下并发读取文件:

• 文件太大，无法加载到内存中
• 线程必须能够异步读取(这是一个网络应用程序)
• 该文件从未被任何线程写入
• 每个线程都将始终知道它需要读取的字节的确切偏移量和长度(即 - 应用程序本身不会“寻找”)。

根据文档 ( https://docs.oracle.com/javase/8/docs/api/java/nio/Buffer.html ) 缓冲区不是线程安全的，因为它们保持内部状态(位置等)。有没有办法在不将文件全部加载到内存的情况下同时随机访问文件？

尽管 FileChannel 在技术上是线程安全的，但来自文档:

Where the file channel is obtained from an existing stream or random access file then the state of the file channel is intimately connected to that of the object whose getChannel method returned the channel. Changing the channel's position, whether explicitly or by reading or writing bytes, will change the file position of the originating object, and vice versa

所以看起来它只是简单地同步。如果我要在每个线程中 new RandomAccessFile().getChannel().map() [编辑:在每次读取时] 那么 MappedByteBuffers 每次读取时不会产生 I/O 开销应该避免？

Answer 1

我不会使用多个线程进行并发读取，而是使用 this approach (基于一个巨大的 CSV 文件的示例，其行必须通过 HTTP 同时发送):

Reading a single file at multiple positions concurrently wouldn't let you go any faster (but it could slow you down considerably).

Instead of reading the file from multiple threads, read the file from a single thread, and parallelize the processing of these lines. A single thread should read your CSV line-by-line, and put each line in a queue. Multiple working threads should then take the next line from the queue, parse it, convert to a request, and process the request concurrently as needed. The splitting of the work would then be done by a single thread, ensuring that there are no missing lines or overlaps.

如果你能逐行读取文件，LineIterator来自 Commons IO是一种内存有效的可能性。如果你必须使用 block ，你的 MappedByteBuffer似乎是一种合理的做法。对于队列，我将使用具有固定容量的阻塞队列，例如 ArrayBlockingQueue — 更好地控制内存使用(队列中的行/ block + 工作线程之间的行/ block = 内存中的行/ block )。