C++ 在 MPI 进程之间共享大型数组和数据结构

Question

我有一个程序目前可以生成大小可达 10GB 的大型数组和矩阵。该程序使用 MPI 来并行化工作负载，但受到以下事实的限制:每个进程都需要自己的数组或矩阵拷贝才能执行其计算部分。内存要求使这个问题无法解决大量 MPI 进程，因此我一直在研究 Boost::Interprocess 作为在 MPI 进程之间共享数据的一种方式。

到目前为止，我已经想出了以下方法，它创建了一个大 vector 并并行化了其元素的总和:

#include <cstdlib>
#include <ctime>
#include <functional>
#include <iostream>
#include <string>
#include <utility>

#include <boost/interprocess/managed_shared_memory.hpp>
#include <boost/interprocess/containers/vector.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/tuple/tuple_comparison.hpp>
#include <mpi.h>

typedef boost::interprocess::allocator<double, boost::interprocess::managed_shared_memory::segment_manager> ShmemAllocator;
typedef boost::interprocess::vector<double, ShmemAllocator> MyVector;

const std::size_t vector_size = 1000000000;
const std::string shared_memory_name = "vector_shared_test.cpp";

int main(int argc, char **argv) {
    int numprocs, rank;

    MPI::Init();
    numprocs = MPI::COMM_WORLD.Get_size();
    rank = MPI::COMM_WORLD.Get_rank();

    if(numprocs >= 2) {
        if(rank == 0) {
            std::cout << "On process rank " << rank << "." << std::endl;
            std::time_t creation_start = std::time(NULL);

            boost::interprocess::shared_memory_object::remove(shared_memory_name.c_str());
            boost::interprocess::managed_shared_memory segment(boost::interprocess::create_only, shared_memory_name.c_str(), size_t(12000000000));

            std::cout << "Size of double: " << sizeof(double) << std::endl;
            std::cout << "Allocated shared memory: " << segment.get_size() << std::endl;

            const ShmemAllocator alloc_inst(segment.get_segment_manager());

            MyVector *myvector = segment.construct<MyVector>("MyVector")(alloc_inst);

            std::cout << "myvector max size: " << myvector->max_size() << std::endl;

            for(int i = 0; i < vector_size; i++) {
                myvector->push_back(double(i));
            }

            std::cout << "Vector capacity: " << myvector->capacity() << " | Memory Free: " << segment.get_free_memory() << std::endl;

            std::cout << "Vector creation successful and took " << std::difftime(std::time(NULL), creation_start) << " seconds." << std::endl;
        }

        std::flush(std::cout);
        MPI::COMM_WORLD.Barrier();

        std::time_t summing_start = std::time(NULL);

        std::cout << "On process rank " << rank << "." << std::endl;
        boost::interprocess::managed_shared_memory segment(boost::interprocess::open_only, shared_memory_name.c_str());

        MyVector *myvector = segment.find<MyVector>("MyVector").first;
        double result = 0;

        for(int i = rank; i < myvector->size(); i = i + numprocs) {
            result = result + (*myvector)[i];
        }
        double total = 0;
        MPI::COMM_WORLD.Reduce(&result, &total, 1, MPI::DOUBLE, MPI::SUM, 0);

        std::flush(std::cout);
        MPI::COMM_WORLD.Barrier();

        if(rank == 0) {
            std::cout << "On process rank " << rank << "." << std::endl;
            std::cout << "Vector summing successful and took " << std::difftime(std::time(NULL), summing_start) << " seconds." << std::endl;

            std::cout << "The arithmetic sum of the elements in the vector is " << total << std::endl;
            segment.destroy<MyVector>("MyVector");
        }

        std::flush(std::cout);
        MPI::COMM_WORLD.Barrier();

        boost::interprocess::shared_memory_object::remove(shared_memory_name.c_str());
    }

    sleep(300);
    MPI::Finalize();

    return 0;
}

我注意到这会导致整个共享对象被映射到每个进程的虚拟内存空间——这是我们计算集群的一个问题，因为它限制虚拟内存与物理内存相同。有没有办法共享这个数据结构而不必映射整个共享内存空间——也许是以共享某种指针的形式？尝试访问未映射的共享内存是否会被定义为行为？不幸的是，我们在数组上执行的操作意味着每个进程最终都需要访问其中的每个元素(尽管不是并发的——我想它可以将共享数组分解成多个部分并为你需要的部分交换数组，但是这并不理想)。

Answer 1

既然你要分享的数据这么大，那么把数据当成一个真正的文件，用文件操作来读取你想要的数据可能更实用。那么，就不需要使用共享内存来共享文件了，让各个进程直接从文件系统中读取即可。

ifstream file ("data.dat", ios::in | ios::binary);
file.seekg(someOffset, ios::beg);
file.read(array, sizeof(array));