c# - 写入内存映射文件比非内存映射文件慢

Question

我正在尝试使用内存映射文件来编写具有高 IO 需求的应用程序。在此应用程序中，我接收到的数据突发速度快于磁盘能够支持的速度。为了避免在我的应用程序中出现缓冲逻辑，我考虑过使用内存映射文件。对于这种文件，我只需将映射到文件的内存写入(比磁盘支持的速度更快)，操作系统最终会将这些数据刷新到磁盘。因此操作系统正在为我做缓冲。

经过实验，我发现内存映射文件使写入内存的速度更快，但刷新到磁盘的速度比普通文件慢。这就是使我得出该结论的原因。这是一段代码，它尽可能快地写入非内存映射文件:

    private static void WriteNonMemoryMappedFile(long fileSize, byte[] bufferToWrite)
    {
        Console.WriteLine(" ==> Non memory mapped file");

        string normalFileName = Path.Combine(Path.GetTempPath(), "MemoryMappedFileWriteTest-NonMmf.bin");
        if (File.Exists(normalFileName))
        {
            File.Delete(normalFileName);
        }

        var stopWatch = Stopwatch.StartNew();
        using (var file = File.OpenWrite(normalFileName))
        {
            var numberOfPages = fileSize/bufferToWrite.Length;

            for (int page = 0; page < numberOfPages; page++)
            {
                file.Write(bufferToWrite, 0, bufferToWrite.Length);
            }
        }

        Console.WriteLine("Non-memory mapped file is now closed after {0} seconds ({1} MB/s)", stopWatch.Elapsed.TotalSeconds, GetSpeed(fileSize, stopWatch));
    }

这段代码的结果是:

==> Non memory mapped file
Non-memory mapped file is now closed after 10.5918587 seconds (966.687541390441 MB/s)

如您所见，我的磁盘速度相当快。这将是我的内存映射文件基准。

现在我尝试使用不安全代码将相同的数据写入内存映射文件(因为这是我打算在我的应用程序中做的):

    [DllImport("msvcrt.dll", EntryPoint = "memcpy", CallingConvention = CallingConvention.Cdecl, SetLastError = false)]
    public static extern IntPtr memcpy(IntPtr dest, IntPtr src, UIntPtr count);

    private static unsafe void WriteMemoryMappedFileWithUnsafeCode(long fileSize, byte[] bufferToWrite)
    {
        Console.WriteLine(" ==> Memory mapped file with unsafe code");

        string fileName = Path.Combine(Path.GetTempPath(), "MemoryMappedFileWriteTest-MmfUnsafeCode.bin");
        if (File.Exists(fileName))
        {
            File.Delete(fileName);
        }

        string mapName = Guid.NewGuid().ToString();

        var stopWatch = Stopwatch.StartNew();
        using (var memoryMappedFile = MemoryMappedFile.CreateFromFile(fileName, FileMode.Create, mapName, fileSize, MemoryMappedFileAccess.ReadWrite))
        using (var view = memoryMappedFile.CreateViewAccessor(0, fileSize, MemoryMappedFileAccess.Write))
        {
            unsafe
            {
                fixed (byte* pageToWritePointer = bufferToWrite)
                {
                    byte* pointer = null;
                    try
                    {
                        view.SafeMemoryMappedViewHandle.AcquirePointer(ref pointer);

                        var writePointer = pointer;

                        var numberOfPages = fileSize/bufferToWrite.Length;

                        for (int page = 0; page < numberOfPages; page++)
                        {
                            memcpy((IntPtr) writePointer, (IntPtr) pageToWritePointer, (UIntPtr) bufferToWrite.Length);
                            writePointer += bufferToWrite.Length;
                        }
                    }
                    finally
                    {
                        if (pointer != null)
                            view.SafeMemoryMappedViewHandle.ReleasePointer();
                    }
                }
            }

            Console.WriteLine("All bytes written in MMF after {0} seconds ({1} MB/s). Will now close MMF. This may be long since some content may not have been flushed to disk yet.", stopWatch.Elapsed.TotalSeconds, GetSpeed(fileSize, stopWatch));
        }

        Console.WriteLine("File is now closed after {0} seconds ({1} MB/s)", stopWatch.Elapsed.TotalSeconds, GetSpeed(fileSize, stopWatch));
    }

然后我得到这个:

==> Memory mapped file with unsafe code
All bytes written in MMF after 6.5442406 seconds (1564.73302033172 MB/s). Will now close MMF. This may be long since some content may not have been flushed to disk yet.
File is now closed after 18.8873186 seconds (542.162704287661 MB/s)

如您所见，这要慢得多。它写入非内存映射文件的大约 56%。

然后我尝试了另一件事。我尝试使用 ViewStreamAccessor 而不是不安全的代码:

    private static unsafe void WriteMemoryMappedFileWithViewStream(long fileSize, byte[] bufferToWrite)
    {
        Console.WriteLine(" ==> Memory mapped file with view stream");
        string fileName = Path.Combine(Path.GetTempPath(), "MemoryMappedFileWriteTest-MmfViewStream.bin");
        if (File.Exists(fileName))
        {
            File.Delete(fileName);
        }

        string mapName = Guid.NewGuid().ToString();

        var stopWatch = Stopwatch.StartNew();
        using (var memoryMappedFile = MemoryMappedFile.CreateFromFile(fileName, FileMode.Create, mapName, fileSize, MemoryMappedFileAccess.ReadWrite))
        using (var viewStream = memoryMappedFile.CreateViewStream(0, fileSize, MemoryMappedFileAccess.Write))
        {
            var numberOfPages = fileSize / bufferToWrite.Length;

            for (int page = 0; page < numberOfPages; page++)
            {
                viewStream.Write(bufferToWrite, 0, bufferToWrite.Length);
            }                

            Console.WriteLine("All bytes written in MMF after {0} seconds ({1} MB/s). Will now close MMF. This may be long since some content may not have been flushed to disk yet.", stopWatch.Elapsed.TotalSeconds, GetSpeed(fileSize, stopWatch));
        }

        Console.WriteLine("File is now closed after {0} seconds ({1} MB/s)", stopWatch.Elapsed.TotalSeconds, GetSpeed(fileSize, stopWatch));
    }

然后我得到这个:

==> Memory mapped file with view stream
All bytes written in MMF after 4.6713875 seconds (2192.06548076352 MB/s). Will now close MMF. This may be long since some content may not have been flushed to disk yet.
File is now closed after 16.8921666 seconds (606.198141569359 MB/s)

再一次，这比非内存映射文件要慢得多。

那么，有没有人知道如何让内存映射文件在写的时候和非内存映射文件一样快？

顺便说一句，这是我的测试程序的剩余部分:

    static void Main(string[] args)
    {
        var bufferToWrite = Enumerable.Range(0, Environment.SystemPageSize * 256).Select(i => (byte)i).ToArray();
        long fileSize = 10 * 1024 * 1024 * 1024L; // 2 GB

        WriteNonMemoryMappedFile(fileSize, bufferToWrite);
        WriteMemoryMappedFileWithUnsafeCode(fileSize, bufferToWrite);
        WriteMemoryMappedFileWithViewStream(fileSize, bufferToWrite);
    }

    private static double GetSpeed(long fileSize, Stopwatch stopwatch)
    {
        var mb = fileSize / 1024.0 / 1024.0;
        var mbPerSec = mb / stopwatch.Elapsed.TotalSeconds;
        return mbPerSec;
    }

编辑 1:

按照 usr 的建议，我尝试使用 SequenceScan 选项。不幸的是，它没有任何影响。这是我所做的更改:

        using (var file = new FileStream(fileName, FileMode.CreateNew, FileAccess.ReadWrite, FileShare.None, 4096, FileOptions.SequentialScan))
        using (var memoryMappedFile = MemoryMappedFile.CreateFromFile(file, mapName, fileSize, MemoryMappedFileAccess.ReadWrite, null, HandleInheritability.None, leaveOpen: false))

Answer 1

来自SDK documentation :

Modified pages in the unmapped view are not written to disk until their share count reaches zero, or in other words, until they are unmapped or trimmed from the working sets of all processes that share the pages. Even then, the modified pages are written "lazily" to disk; that is, modifications may be cached in memory and written to disk at a later time. To minimize the risk of data loss in the event of a power failure or a system crash, applications should explicitly flush modified pages using the FlushViewOfFile function.

.NET 程序员认真对待最后一句话，MemoryMappedViewStream.Dispose() method您调用的实际上调用了 FlushViewOfFile()。这需要时间，您会在个人资料结果中看到这一点。技术上可以绕过此调用，不要调用 Dispose() 并让终结器关闭 View 句柄。

FileStream 不对文件执行等效操作 (FlushFileBuffers)，因此您可以获得从文件系统缓存到磁盘的延迟写入的全部好处。在 Dispose() 调用后很长时间内发生，您的程序无法观察到。