python - 在 MPI Python 程序中查找 SegFault 的一般建议

Question

我使用 mpi4py 库使用 Python 编写 MPI 程序。 Python 生成并调用低级 C 代码。

我刚刚遇到了段错误，需要对其进行追踪。

如果我直接用 C 语言编写，我会在 gdb 中编译并运行我的程序，找到有问题的代码行，然后从那里开始调试。

如果我用Python编写，我仍然会使用gdb(您可以调用gdb python filename.py，gdb会将您指向C的有问题的行。)

但是可惜我正在使用 MPI，它使用 Python，而 Python 使用 C。我得到这样的堆栈跟踪

mrocklin@baconost:~/workspace/ape$ mpiexec -np 3 -hostfile tmp/hostfile -rankfile tmp/rankfile python ape/codegen/run.py tmp/
[baconost:27370] *** Process received signal ***
[baconost:27370] Signal: Segmentation fault (11)
[baconost:27370] Signal code: Address not mapped (1)
[baconost:27370] Failing at address: 0x8
[baconost:27370] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) [0x7fcd529c5c60]
[baconost:27370] [ 1] /home/mrocklin/Software/openmpi/lib/libmpi.so.0(MPI_Comm_get_errhandler+0xa0) [0x7fcd46fae240]
[baconost:27370] [ 2] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/python2.7/site-packages/mpi4py/MPI.so(+0x2ddbc) [0x7fcd47234dbc]
[baconost:27370] [ 3] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/python2.7/site-packages/mpi4py/MPI.so(initMPI+0x1f2e) [0x7fcd4724f95e]
[baconost:27370] [ 4] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(_PyImport_LoadDynamicModule+0xc2) [0x7fcd52cc2e02]
[baconost:27370] [ 5] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xecd90) [0x7fcd52cc0d90]
[baconost:27370] [ 6] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xed031) [0x7fcd52cc1031]
[baconost:27370] [ 7] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyImport_ImportModuleLevel+0x2be) [0x7fcd52cc206e]
[baconost:27370] [ 8] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xd3fed) [0x7fcd52ca7fed]
[baconost:27370] [ 9] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyObject_Call+0x68) [0x7fcd52c19d18]
[baconost:27370] [10] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x56) [0x7fcd52ca8516]
[baconost:27370] [11] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x28b8) [0x7fcd52caba78]
[baconost:27370] [12] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8d2) [0x7fcd52cb0722]
[baconost:27370] [13] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32) [0x7fcd52cb0772]
[baconost:27370] [14] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0xc2) [0x7fcd52cbf6e2]
[baconost:27370] [15] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xebcae) [0x7fcd52cbfcae]
[baconost:27370] [16] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xecd90) [0x7fcd52cc0d90]
[baconost:27370] [17] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xed264) [0x7fcd52cc1264]
[baconost:27370] [18] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyImport_ImportModuleLevel+0x118) [0x7fcd52cc1ec8]
[baconost:27370] [19] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xd3fed) [0x7fcd52ca7fed]
[baconost:27370] [20] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyObject_Call+0x68) [0x7fcd52c19d18]
[baconost:27370] [21] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x56) [0x7fcd52ca8516]
[baconost:27370] [22] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x28b8) [0x7fcd52caba78]
[baconost:27370] [23] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8d2) [0x7fcd52cb0722]
[baconost:27370] [24] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32) [0x7fcd52cb0772]
[baconost:27370] [25] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0xc2) [0x7fcd52cbf6e2]
[baconost:27370] [26] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xebcae) [0x7fcd52cbfcae]
[baconost:27370] [27] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xed58d) [0x7fcd52cc158d]
[baconost:27370] [28] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xecd90) [0x7fcd52cc0d90]
[baconost:27370] [29] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xed264) [0x7fcd52cc1264]
[baconost:27370] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 27370 on node baconost.cs.uchicago.edu exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

两个问题:

1. 在一般情况下，在检测到错误后使用此软件堆栈调试程序的最佳方法是什么(我想避免“您应该通过测试来防止错误而不是诊断错误”的评论。我同意这一点，但尽管我们尽了最大努力，有时还是会发生段错误)

2. 在这种特定情况下，这是否表明我遇到了 MPI/mpi4py 问题？最后几行表明情况可能就是这样。

Answer 1

此特定错误的问题是旧版本的 OpenMPI 不能很好地处理排名文件。此错误已报告，并且似乎已从 1.6.2 开始修复。显然这部分代码很少使用且未经测试。