linux - MPI_Comm_size 段错误

Question

嗯，大家好。我在 Linux 中使用 MPI 和 OpenMP 运行并行程序时遇到这些错误，

 [node65:03788] *** Process received signal ***
 [node65:03788] Signal: Segmentation fault (11)
 [node65:03788] Signal code: Address not mapped (1)
 [node65:03788] Failing at address: 0x44000098
 [node65:03788] [ 0] /lib64/libpthread.so.0 [0x2b663e446c00]
 [node65:03788] [ 1] /public/share/mpi/openmpi-   1.4.5//lib/libmpi.so.0(MPI_Comm_size+0x60) [0x2b663d694360]
 [node65:03788] [ 2] fdtd_3D_xyzPML_MPI_OpenMP(main+0xaa) [0x42479a]
 [node65:03788] [ 3] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b663e56f184]
 [node65:03788] [ 4] fdtd_3D_xyzPML_MPI_OpenMP(_ZNSt8ios_base4InitD1Ev+0x39) [0x405d79]
 [node65:03788] *** End of error message ***
 -----------------------------------------------------------------------------
 mpirun noticed that process rank 2 with PID 3787 on node node65 exited on signal 11 (Segmentation fault).
 -----------------------------------------------------------------------------

分析核心文件后，我得到以下信息:

[Thread debugging using libthread_db enabled]
[New Thread 47310344057648 (LWP 26962)]
[New Thread 1075841344 (LWP 26966)]
[New Thread 1077942592 (LWP 26967)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 47310344057648 (LWP 26962)]
0x00002b074afb3360 in PMPI_Comm_size () from /public/share/mpi/openmpi-1.4.5//lib/libmpi.so.0

是什么原因造成的？感谢您的帮助

代码(test.cpp)如下，大家可以试试看:

#include <stdio.h> 
#include <stdlib.h>
#include <omp.h>
#include "mpi.h"

int main(int argc, char* argv[])
{
int nprocs = 1; //the number of processes
int myrank = 0; 
int provide;

MPI_Init_thread(&argc,&argv,MPI_THREAD_FUNNELED,&provide);
if (MPI_THREAD_FUNNELED != provide)  
{  
    printf ("%d != required %d", MPI_THREAD_FUNNELED, provide);  
    return 0;  
}

MPI_Comm_size(MPI_COMM_WORLD,&nprocs); 
MPI_Comm_rank(MPI_COMM_WORLD,&myrank);  

int num_threads = 1;      //Openmp
omp_set_dynamic(1);
num_threads = 16;
omp_set_num_threads(num_threads);

#pragma omp parallel  
{  
    printf ("%d omp thread from %d mpi process\n", omp_get_thread_num(), myrank);  

}  
MPI_Finalize();

}

Answer 1

嗯，这可能不多，甚至有点蹩脚的答案，但我在混合不同的 MPI 安装(准确地说是 OpenMPI 和 MVAPICH2)时遇到了这个问题。

这里有一些要检查的东西

• 针对您链接的 MPI 版本

ldd <application> | grep -i mpi
    libmpi.so.1 => /usr/lib64/mpi/gcc/openmpi/lib64/libmpi.so.1 (0x00007f90c03cc000)

• 动态加载的 MPI 版本

echo $LD_LIBRARY_PATH | tr : "\n" | grep -i mpi
/usr/lib64/mpi/gcc/openmpi/lib64

• 是否覆盖这个动态加载(这个变量应该是空的，除非你知道你在做什么)

echo $LD_PRELOAD 

如果一切正常，您需要检查您链接到的每个依赖 MPI 的库是否也链接到相同的版本。如果没有其他库链接到 MPI，则不会出现任何内容。

ldd <application> | sed "s/^\s*\(.*=> \)\?//;s/ (0x[0-9a-fA-F]*)$//" | xargs -L 1 ldd | grep -i mpi

如果确实出现了可疑的东西，请说 libmpich.so.3 =>/usr/lib64/mp​​i/gcc/MVAPICH2/1.8.1/lib/libmpich.so.3 例如，您应该删除 -L 1 并将 grep 替换为可视化的内容(nothing ? 或 less，或 vim - ...)，然后搜索该可疑行。