linux 内存中的软锁定

Question

My system is a embedded linux system(running kernel version 2.6.18). A client process send data to mysql server. The data will be stored in mysql database at a RAID5 assembled by four disks. The IO pressure(wa%) is always above 20% , mysql CPU utilization is very high.

运行 5 或 6 小时后，系统遇到 softlock up stat。 堆栈信息是关于释放物理内存、将缓存数据写入硬盘的信息。

在这种情况下有什么建议吗？**

BUG: soft lockup detected on CPU#0!
 [<c043dc1c>] softlockup_tick+0x8f/0xb1
 [<c0428cb5>] update_process_times+0x26/0x5c
 [<c0411256>] smp_apic_timer_interrupt+0x5d/0x67
 [<c04044e7>] apic_timer_interrupt+0x1f/0x24
 [<c06fe0b9>] _spin_lock+0x5/0xf
 [<c047db2a>] __mark_inode_dirty+0x50/0x176
 [<c0424eef>] current_fs_time+0x4d/0x5e
 [<c0475ccd>] touch_atime+0x51/0x94
 [<c0440926>] do_generic_mapping_read+0x425/0x563
 [<c044134b>] __generic_file_aio_read+0xf3/0x267
 [<c043fcd0>] file_read_actor+0x0/0xd4
 [<c04414fb>] generic_file_aio_read+0x3c/0x4d
 [<c045d72d>] do_sync_read+0xc1/0xfd
 [<c0431656>] autoremove_wake_function+0x0/0x37
 [<c045e0e8>] vfs_read+0xa4/0x167
 [<c045d66c>] do_sync_read+0x0/0xfd
 [<c045e688>] sys_pread64+0x5e/0x62
 [<c0403a27>] syscall_call+0x7/0xb
 =======================
BUG: soft lockup detected on CPU#2!
 [<c043dc1c>] softlockup_tick+0x8f/0xb1
 [<c0428cb5>] update_process_times+0x26/0x5c
 [<c0411256>] smp_apic_timer_interrupt+0x5d/0x67
 [<c04044e7>] apic_timer_interrupt+0x1f/0x24
 [<c06fe0bb>] _spin_lock+0x7/0xf
 [<c04aaf17>] journal_try_to_free_buffers+0xf4/0x17b
 [<c0442c52>] find_get_pages+0x28/0x5d
 [<c049c4b1>] ext3_releasepage+0x0/0x7d
 [<c045f0bf>] try_to_release_page+0x2c/0x46
 [<c0447894>] invalidate_mapping_pages+0xc9/0x167
 [<c04813b0>] drop_pagecache+0x86/0xd2
 [<c048144e>] drop_caches_sysctl_handler+0x52/0x64
 [<c04813fc>] drop_caches_sysctl_handler+0x0/0x64
 [<c042623d>] do_rw_proc+0xe8/0xf4
 [<c0426268>] proc_writesys+0x1f/0x24
 [<c045df81>] vfs_write+0xa6/0x169
 [<c0426249>] proc_writesys+0x0/0x24
 [<c045e601>] sys_write+0x41/0x6a
 [<c0403a27>] syscall_call+0x7/0xb
 =======================
BUG: soft lockup detected on CPU#1!
 [<c043dc1c>] softlockup_tick+0x8f/0xb1
 [<c0428cb5>] update_process_times+0x26/0x5c
 [<c0411256>] smp_apic_timer_interrupt+0x5d/0x67
 [<c04044e7>] apic_timer_interrupt+0x1f/0x24
 [<c06f007b>] inet_diag_dump+0x804/0x821
 [<c06fe0bb>] _spin_lock+0x7/0xf
 [<c047db2a>] __mark_inode_dirty+0x50/0x176
 [<c043168d>] wake_bit_function+0x0/0x3c
 [<c04ae0f6>] __journal_remove_journal_head+0xee/0x1a5
 [<c0445ae8>] __set_page_dirty_nobuffers+0x87/0xc6
 [<c04a908e>] __journal_unfile_buffer+0x8/0x11
 [<c04ab94d>] journal_commit_transaction+0x8e0/0x1103
 [<c0431656>] autoremove_wake_function+0x0/0x37
 [<c04af690>] kjournald+0xa9/0x1e5
 [<c0431656>] autoremove_wake_function+0x0/0x37
 [<c04af5e7>] kjournald+0x0/0x1e5
 [<c04314da>] kthread+0xde/0xe2
 [<c04313fc>] kthread+0x0/0xe2
 [<c0404763>] kernel_thread_helper+0x7/0x14
 =======================
BUG: soft lockup detected on CPU#3!
 [<c043dc1c>] softlockup_tick+0x8f/0xb1
 [<c0428cb5>] update_process_times+0x26/0x5c
 [<c0411256>] smp_apic_timer_interrupt+0x5d/0x67
 [<c04044e7>] apic_timer_interrupt+0x1f/0x24
 [<c06fe0bb>] _spin_lock+0x7/0xf
 [<c047db2a>] __mark_inode_dirty+0x50/0x176
 [<c0424eef>] current_fs_time+0x4d/0x5e
 [<c0475ccd>] touch_atime+0x51/0x94
 [<c0440926>] do_generic_mapping_read+0x425/0x563
 [<c044134b>] __generic_file_aio_read+0xf3/0x267
 [<c043fcd0>] file_read_actor+0x0/0xd4
 [<c04414fb>] generic_file_aio_read+0x3c/0x4d
 [<c045d72d>] do_sync_read+0xc1/0xfd
 [<c0431656>] autoremove_wake_function+0x0/0x37
 [<c045e0e8>] vfs_read+0xa4/0x167
 [<c045d66c>] do_sync_read+0x0/0xfd
 [<c045e688>] sys_pread64+0x5e/0x62
 [<c0403a27>] syscall_call+0x7/0xb

========================

Answer 1

说真的，尝试一些新的东西。 2.6.18 已超过 7 岁。