c++ - Boost::进程间共享内存总线错误

标签 c++ linux boost shared-memory boost-interprocess

我在使用 Open-MPI 1.3.3 的集群上使用 CentOS 5.4 x86_64 和 Boost 1.42.0。我正在编写一个共享库，它使用共享内存来存储大量数据以供多个进程使用。还有一个加载程序应用程序将从文件中读取数据并将其加载到共享内存中。

当我运行加载应用程序时，它会确定准确存储数据所需的内存量，然后添加 25% 的开销。几乎每个文件的数据量都超过 2 GB。当我使用 Boost 的进程间库发出内存请求时，它表示它已成功保留所请求的内存量。但是当我使用开始使用它时，我收到“总线错误”。据我所知，总线错误是由于访问内存段可用范围之外的内存而导致的。

因此，我开始研究共享内存在 Linux 上的情况，以及需要检查哪些内容以确保我的系统正确配置为允许大量共享内存。

  1. 我查看了 /proc/sys/kernel/shm* 中的"file":
    • shmall - 4294967296(4 GB)
    • shmmax - 68719476736(68 GB)
    • shmmni - 4096

  2. 我调用了 ipcs -lm 命令: 
    ------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 67108864
max total shared memory (kbytes) = 17179869184
min seg size (bytes) = 1

From what I can tell, those settings indicate that I should be able to allocate enough shared memory for my purposes. So I created a stripped down program that created large amounts of data in shared memory:


#include <iostream>

#include <boost/interprocess/managed_shared_memory.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/containers/vector.hpp>

namespace bip = boost::interprocess;

typedef bip::managed_shared_memory::segment_manager segment_manager_t;
typedef bip::allocator<long, segment_manager_t> long_allocator;
typedef bip::vector<long, long_allocator> long_vector;

int main(int argc, char ** argv) {
    struct shm_remove  { 
        shm_remove()    { bip::shared_memory_object::remove("ShmTest"); } 
        ~shm_remove()   { bip::shared_memory_object::remove("ShmTest"); } 
    } remover; 

    size_t szLength = 280000000;
    size_t szRequired = szLength * sizeof(long);
    size_t szRequested = (size_t) (szRequired * 1.05);
    bip::managed_shared_memory segment(bip::create_only, "ShmTest", szRequested); 

    std::cout << 
        "Length:       " << szLength << "\n" <<
        "sizeof(long): " << sizeof(long) << "\n" <<
        "Required:     " << szRequired << "\n" <<
        "Requested:    " << szRequested << "\n" <<
        "Allocated:    " << segment.get_size() << "\n" <<
        "Overhead:     " << segment.get_size() - segment.get_free_memory() << "\n" <<
        "Free:         " << segment.get_free_memory() << "\n\n";

    long_allocator alloc(segment.get_segment_manager()); 
    long_vector vector(alloc);

    if (argc > 1) {
        std::cout << "Reserving Length of " << szLength << "\n";
        vector.reserve(szLength);
        std::cout << "Vector Capacity: " << vector.capacity() << "\tFree: " << segment.get_free_memory() << "\n\n";
    }

    for (size_t i = 0; i < szLength; i++) {
        if ((i % (szLength / 100)) == 0) {
            std::cout << i << ": " << "\tVector Capacity: " << vector.capacity() << "\tFree: " << segment.get_free_memory() << "\n";
        }
        vector.push_back(i);    
    }
    std::cout << "end: " << "\tVector Capacity: " << vector.capacity() << "\tFree: " << segment.get_free_memory() << "\n";

    return 0;
}

使用以下行编译它:

g++ ShmTest.cpp -lboost_system -lrt

然后使用以下输出运行它(经过编辑以使其更小):

Length:       280000000
sizeof(long): 8
Required:     2240000000
Requested:    2352000000
Allocated:    2352000000
Overhead:     224
Free:         2351999776

0:      Vector Capacity: 0      Free: 2351999776
2800000:        Vector Capacity: 3343205        Free: 2325254128
5600000:        Vector Capacity: 8558607        Free: 2283530912
8400000:        Vector Capacity: 8558607        Free: 2283530912
11200000:       Vector Capacity: 13693771       Free: 2242449600
14000000:       Vector Capacity: 21910035       Free: 2176719488
...
19600000:       Vector Capacity: 21910035       Free: 2176719488
22400000:       Vector Capacity: 35056057       Free: 2071551312
...
33600000:       Vector Capacity: 35056057       Free: 2071551312
36400000:       Vector Capacity: 56089691       Free: 1903282240
...
56000000:       Vector Capacity: 56089691       Free: 1903282240
58800000:       Vector Capacity: 89743507       Free: 1634051712
...
89600000:       Vector Capacity: 89743507       Free: 1634051712
92400000:       Vector Capacity: 143589611      Free: 1203282880
...
142800000:      Vector Capacity: 143589611      Free: 1203282880
145600000:      Vector Capacity: 215384417      Free: 628924432
...
212800000:      Vector Capacity: 215384417      Free: 628924432
215600000:      Vector Capacity: 293999969      Free: 16
...
260400000:      Vector Capacity: 293999969      Free: 16
Bus error

如果您使用 a 参数运行程序(任何参数都可以，只需增加 argc )，它会预分配 vector ，但仍会在同一数组索引处导致总线错误。

我使用 /dev/shm 命令检查了 ls -ash /dev/shm 中"file"的大小:

total 2.0G
   0 .     0 ..  2.0G ShmTest

就像我原来的应用程序一样，分配的共享内存的大小上限为 2 GB。鉴于它“成功”分配了 2352000000 字节的内存，以 GB 为单位(使用 1024*1024*1024)，它应该是 2.19 Gb。

当我运行实际程序以使用 MPI 加载数据时，我收到以下错误输出:

Requested: 2808771120
Recieved: 2808771120

[c1-master:13894] *** Process received signal ***
[c1-master:13894] Signal: Bus error (7)
[c1-master:13894] Signal code:  (2)
[c1-master:13894] Failing at address: 0x2b3190157000
[c1-master:13894] [ 0] /lib64/libpthread.so.0 [0x3a64e0e7c0]
[c1-master:13894] [ 1] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost12interprocess26uninitialized_copy_or_moveINS0_10offset_ptrIlEEPlEET0_T_S6_S5_PNS_10disable_ifINS0_11move_detail16is_move_iteratorIS6_EEvE4typeE+0x218) [0x2b310dcf3fb8]
[c1-master:13894] [ 2] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost9container6vectorIlNS_12interprocess9allocatorIlNS2_15segment_managerIcNS2_15rbtree_best_fitINS2_12mutex_familyENS2_10offset_ptrIvEELm0EEENS2_10iset_indexEEEEEE15priv_assign_auxINS7_IlEEEEvT_SG_St20forward_iterator_tag+0xa75) [0x2b310dd0a335]
[c1-master:13894] [ 3] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost9container17containers_detail25advanced_insert_aux_proxyINS0_6vectorIlNS_12interprocess9allocatorIlNS4_15segment_managerIcNS4_15rbtree_best_fitINS4_12mutex_familyENS4_10offset_ptrIvEELm0EEENS4_10iset_indexEEEEEEENS0_17constant_iteratorISF_lEEPSF_E25uninitialized_copy_all_toESI_+0x1d7) [0x2b310dd0b817]
[c1-master:13894] [ 4] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost9container6vectorINS1_IlNS_12interprocess9allocatorIlNS2_15segment_managerIcNS2_15rbtree_best_fitINS2_12mutex_familyENS2_10offset_ptrIvEELm0EEENS2_10iset_indexEEEEEEENS3_ISD_SB_EEE17priv_range_insertENS7_ISD_EEmRNS0_17containers_detail23advanced_insert_aux_intISD_PSD_EE+0x771) [0x2b310dd0d521]
[c1-master:13894] [ 5] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost12interprocess6detail8Ctor3ArgINS_9container6vectorINS4_IlNS0_9allocatorIlNS0_15segment_managerIcNS0_15rbtree_best_fitINS0_12mutex_familyENS0_10offset_ptrIvEELm0EEENS0_10iset_indexEEEEEEENS5_ISF_SD_EEEELb0EiSF_NS5_IvSD_EEE11construct_nEPvmRm+0x157) [0x2b310dd0d9a7]
[c1-master:13894] [ 6] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost12interprocess15segment_managerIcNS0_15rbtree_best_fitINS0_12mutex_familyENS0_10offset_ptrIvEELm0EEENS0_10iset_indexEE28priv_generic_named_constructIcEEPvmPKT_mbbRNS0_6detail18in_place_interfaceERNS7_INSE_12index_configISB_S6_EEEENSE_5bool_ILb1EEE+0x6fd) [0x2b310dd0c85d]
[c1-master:13894] [ 7] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost12interprocess15segment_managerIcNS0_15rbtree_best_fitINS0_12mutex_familyENS0_10offset_ptrIvEELm0EEENS0_10iset_indexEE22priv_generic_constructEPKcmbbRNS0_6detail18in_place_interfaceE+0xf8) [0x2b310dd0dd58]
[c1-master:13894] [ 8] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN7POP_LTL16ExportPopulation22InitializeSharedMemoryEPKc+0x1609) [0x2b310dceea99]
[c1-master:13894] [ 9] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN7POP_LTL10InitializeEPKc+0x349) [0x2b310dd0ebb9]
[c1-master:13894] [10] MPI_Release/LookupPopulation.MpiLoader(main+0x372) [0x4205d2]
[c1-master:13894] [11] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3a6461d994]
[c1-master:13894] [12] MPI_Release/LookupPopulation.MpiLoader(__gxx_personality_v0+0x239) [0x420009]
[c1-master:13894] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 13894 on node c1-master exited on signal 7 (Bus error).
--------------------------------------------------------------------------

我真的不知道该去哪里。有人对尝试什么有什么建议吗？

<小时/>

发布到 Boost 错误跟踪:https://svn.boost.org/trac/boost/ticket/4374

最佳答案

好吧，如果你继续寻找答案足够长的时间......

在 Linux 上，它使用的共享内存机制 (tmpfs) 默认将其限制为系统 RAM 的一半。所以在我的集群上它是 2 Gb，因为我们有 4 Gb 系统 RAM。因此，当它尝试分配共享内存段时，它会分配到 /dev/shm 上剩余的最大大小。

但是，当 Boost 库无法分配所请求的内存量时，它没有指示错误，甚至没有报告正确的可用内存量，从而出现了问题。它显然很高兴能够顺利进行，直到到达该段的末尾，然后出错了。

长期解决方案是更新 /etc/fstab 文件以永久进行更改，但可以运行命令行调用来增加每个节点上可用共享内存的大小，直到重新启动。

mount -o 重新挂载，size=XXX/dev/shm

其中 XXX 是可用内存量(例如 size=4G)。

这是从http://www.cyberciti.biz/tips/what-is-devshm-and-its-practical-usage.html中找出/获取的

关于c++ - Boost::进程间共享内存总线错误，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/3103255/

上一篇：java - Android Firebase 数据库路径无效

下一篇：rxjs - 完成后我应该取消订阅吗？

相关文章：

c++ - boost qi 属性出现为 qi::unused_type

c++ - 生成时CMake错误。项目使用Boost库

c++ - 包含来自另一个文件的运算符重载

c++ - 我怎样才能为一个概念重载一个函数？

java - 当我从Java代码执行C++时未显示任何输出

linux - 脚本持续运行数小时

linux - 为什么只有在按回车后才能在串行连接中接收到字符？

linux - Htop process viewer - 设置列宽

c++ - 使用 Cmake-gui 和 vcpkg

c++ - 在 memset 等上使用结构初始化 ({...}) 是否更可取/可以？

©2024 IT工具网  联系我们