我的应用程序在 Tomcat 上运行，一段时间后我发现该进程被 Kernal 终止，我想知道如何调试这个问题。
我在路径/var/log 的 kern.log 文件中看到以下消息:
Jun 24 03:40:48 test-host-name kernel: [4329150.480689] splunkd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Jun 24 03:40:48 test-host-name kernel: [4329150.482610] splunkd cpuset=/ mems_allowed=0
Jun 24 03:40:48 test-host-name kernel: [4329150.488425] CPU: 1 PID: 1619 Comm: splunkd Not tainted 3.19.0-42-generic #48~14.04.1-Ubuntu
Jun 24 03:40:48 test-host-name kernel: [4329150.488431] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
Jun 24 03:40:48 test-host-name kernel: [4329150.488435] 0000000000000000 ffff8800bb9eb888 ffffffff817afcd5 00000000000087ee
Jun 24 03:40:48 test-host-name kernel: [4329150.488443] ffff8802322475c0 ffff8800bb9eb928 ffffffff817aabb7 0000000000000000
Jun 24 03:40:48 test-host-name kernel: [4329150.488448] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Jun 24 03:40:48 test-host-name kernel: [4329150.488455] Call Trace:
Jun 24 03:40:48 test-host-name kernel: [4329150.489426] [<ffffffff817afcd5>] dump_stack+0x45/0x57
Jun 24 03:40:48 test-host-name kernel: [4329150.489433] [<ffffffff817aabb7>] dump_header+0x7f/0x1f1
Jun 24 03:40:48 test-host-name kernel: [4329150.489900] [<ffffffff81179b3b>] oom_kill_process+0x22b/0x390
Jun 24 03:40:48 test-host-name kernel: [4329150.489908] [<ffffffff8117a332>] out_of_memory+0x4d2/0x520
Jun 24 03:40:48 test-host-name kernel: [4329150.489914] [<ffffffff8117f720>] __alloc_pages_nodemask+0x940/0xa60
Jun 24 03:40:48 test-host-name kernel: [4329150.489922] [<ffffffff811c44b1>] alloc_pages_current+0x91/0x100
Jun 24 03:40:48 test-host-name kernel: [4329150.489927] [<ffffffff81176447>] __page_cache_alloc+0xa7/0xc0
Jun 24 03:40:48 test-host-name kernel: [4329150.489931] [<ffffffff811787a8>] filemap_fault+0x1a8/0x430
Jun 24 03:40:48 test-host-name kernel: [4329150.489937] [<ffffffff811a14d9>] __do_fault+0x39/0x90
Jun 24 03:40:48 test-host-name kernel: [4329150.489941] [<ffffffff811a3e67>] do_read_fault.isra.55+0x1d7/0x2f0
Jun 24 03:40:48 test-host-name kernel: [4329150.490489] [<ffffffff810998aa>] ? dequeue_task+0x5a/0x80
Jun 24 03:40:48 test-host-name kernel: [4329150.490498] [<ffffffff811a5a24>] handle_mm_fault+0x604/0x10e0
Jun 24 03:40:48 test-host-name kernel: [4329150.490520] [<ffffffff810dd9e8>] ? hrtimer_try_to_cancel+0x48/0xe0
Jun 24 03:40:48 test-host-name kernel: [4329150.490531] [<ffffffff81061e84>] __do_page_fault+0x1c4/0x5b0
Jun 24 03:40:48 test-host-name kernel: [4329150.492741] [<ffffffff81231511>] ? ep_poll+0x311/0x350
Jun 24 03:40:48 test-host-name kernel: [4329150.492750] [<ffffffff810a0b40>] ? wake_up_state+0x20/0x20
Jun 24 03:40:48 test-host-name kernel: [4329150.492756] [<ffffffff810622a1>] do_page_fault+0x31/0x70
Jun 24 03:40:48 test-host-name kernel: [4329150.492764] [<ffffffff817b9868>] page_fault+0x28/0x30
Jun 24 03:40:48 test-host-name kernel: [4329150.492767] Mem-Info:
Jun 24 03:40:48 test-host-name kernel: [4329150.492771] Node 0 DMA per-cpu:
Jun 24 03:40:48 test-host-name kernel: [4329150.492774] CPU 0: hi: 0, btch: 1 usd: 0
Jun 24 03:40:48 test-host-name kernel: [4329150.492777] CPU 1: hi: 0, btch: 1 usd: 0
Jun 24 03:40:48 test-host-name kernel: [4329150.492781] CPU 2: hi: 0, btch: 1 usd: 0
Jun 24 03:40:48 test-host-name kernel: [4329150.492782] Node 0 DMA32 per-cpu:
Jun 24 03:40:48 test-host-name kernel: [4329150.492785] CPU 0: hi: 186, btch: 31 usd: 20
Jun 24 03:40:48 test-host-name kernel: [4329150.492787] CPU 1: hi: 186, btch: 31 usd: 145
Jun 24 03:40:48 test-host-name kernel: [4329150.492790] CPU 2: hi: 186, btch: 31 usd: 71
Jun 24 03:40:48 test-host-name kernel: [4329150.492791] Node 0 Normal per-cpu:
Jun 24 03:40:48 test-host-name kernel: [4329150.493650] CPU 0: hi: 186, btch: 31 usd: 30
Jun 24 03:40:48 test-host-name kernel: [4329150.493671] CPU 1: hi: 186, btch: 31 usd: 0
Jun 24 03:40:48 test-host-name kernel: [4329150.493673] CPU 2: hi: 186, btch: 31 usd: 0
Jun 24 03:40:48 test-host-name kernel: [4329150.493679] active_anon:1298444 inactive_anon:304238 isolated_anon:0
Jun 24 03:40:48 test-host-name kernel: [4329150.493679] active_file:34 inactive_file:21 isolated_file:0
Jun 24 03:40:48 test-host-name kernel: [4329150.493679] unevictable:0 dirty:5 writeback:0 unstable:0
Jun 24 03:40:48 test-host-name kernel: [4329150.493679] free:22359 slab_reclaimable:3817 slab_unreclaimable:4113
Jun 24 03:40:48 test-host-name kernel: [4329150.493679] mapped:21 shmem:421 pagetables:4681 bounce:0
Jun 24 03:40:48 test-host-name kernel: [4329150.493679] free_cma:0
Jun 24 03:40:48 test-host-name kernel: [4329150.493683] Node 0 DMA free:15900kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jun 24 03:40:48 test-host-name kernel: [4329150.493692] lowmem_reserve[]: 0 2976 7966 7966
Jun 24 03:40:48 test-host-name kernel: [4329150.493697] Node 0 DMA32 free:41900kB min:25196kB low:31492kB high:37792kB active_anon:1842924kB inactive_anon:538140kB active_file:84kB inactive_file:104kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129280kB managed:3049516kB mlocked:0kB dirty:12kB writeback:0kB mapped:0kB shmem:1668kB slab_reclaimable:3816kB slab_unreclaimable:4368kB kernel_stack:960kB pagetables:6072kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:30156 all_unreclaimable? yes
Jun 24 03:40:48 test-host-name kernel: [4329150.493704] lowmem_reserve[]: 0 0 4990 4990
Jun 24 03:40:48 test-host-name kernel: [4329150.493707] Node 0 Normal free:31636kB min:42252kB low:52812kB high:63376kB active_anon:3350852kB inactive_anon:678812kB active_file:52kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:5242880kB managed:5110660kB mlocked:0kB dirty:8kB writeback:0kB mapped:84kB shmem:16kB slab_reclaimable:11452kB slab_unreclaimable:12076kB kernel_stack:2448kB pagetables:12652kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:27348 all_unreclaimable? yes
Jun 24 03:40:48 test-host-name kernel: [4329150.493713] lowmem_reserve[]: 0 0 0 0
Jun 24 03:40:48 test-host-name kernel: [4329150.493717] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
Jun 24 03:40:48 test-host-name kernel: [4329150.493731] Node 0 DMA32: 1832*4kB (UM) 1350*8kB (UM) 679*16kB (UEM) 248*32kB (UM) 78*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41920kB
Jun 24 03:40:48 test-host-name kernel: [4329150.493743] Node 0 Normal: 7929*4kB (M) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 31716kB
Jun 24 03:40:48 test-host-name kernel: [4329150.493753] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun 24 03:40:48 test-host-name kernel: [4329150.493755] 152611 total pagecache pages
Jun 24 03:40:48 test-host-name kernel: [4329150.493757] 152133 pages in swap cache
Jun 24 03:40:48 test-host-name kernel: [4329150.493759] Swap cache stats: add 2429803, delete 2277670, find 32042376/32159653
Jun 24 03:40:48 test-host-name kernel: [4329150.493763] Free swap = 0kB
Jun 24 03:40:48 test-host-name kernel: [4329150.493765] Total swap = 974844kB
Jun 24 03:40:48 test-host-name kernel: [4329150.493767] 2097038 pages RAM
Jun 24 03:40:48 test-host-name kernel: [4329150.493768] 0 pages HighMem/MovableOnly
Jun 24 03:40:48 test-host-name kernel: [4329150.493770] 53017 pages reserved
Jun 24 03:40:48 test-host-name kernel: [4329150.493771] 0 pages cma reserved
Jun 24 03:40:48 test-host-name kernel: [4329150.493774] 0 pages hwpoisoned
Jun 24 03:40:48 test-host-name kernel: [4329150.493776] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Jun 24 03:40:48 test-host-name kernel: [4329150.493801] [ 468] 0 468 3820 3 14 53 0 upstart-file-br
Jun 24 03:40:48 test-host-name kernel: [4329150.493806] [ 501] 101 501 78605 3496 63 2172 0 rsyslogd
Jun 24 03:40:48 test-host-name kernel: [4329150.493810] [ 633] 0 633 3816 18 14 45 0 upstart-socket-
Jun 24 03:40:48 test-host-name kernel: [4329150.493816] [ 637] 0 637 4827 28 14 54 0 rpcbind
Jun 24 03:40:48 test-host-name kernel: [4329150.493821] [ 639] 102 639 24886 48 49 195 0 dbus-daemon
Jun 24 03:40:48 test-host-name kernel: [4329150.493825] [ 777] 0 777 2372 2 10 30 0 getty
Jun 24 03:40:48 test-host-name kernel: [4329150.493831] [ 782] 0 782 2372 2 9 29 0 getty
Jun 24 03:40:48 test-host-name kernel: [4329150.493836] [ 790] 0 790 2372 2 10 32 0 getty
Jun 24 03:40:48 test-host-name kernel: [4329150.493840] [ 793] 0 793 2372 2 11 30 0 getty
Jun 24 03:40:48 test-host-name kernel: [4329150.493848] [ 795] 0 795 2372 2 10 32 0 getty
Jun 24 03:40:48 test-host-name kernel: [4329150.494726] [ 829] 0 829 3202 5 11 31 0 atd
Jun 24 03:40:48 test-host-name kernel: [4329150.494732] [ 831] 0 831 4331 26 14 32 0 cron
Jun 24 03:40:48 test-host-name kernel: [4329150.494736] [ 834] 0 834 1093 2 8 35 0 acpid
Jun 24 03:40:48 test-host-name kernel: [4329150.494742] [ 910] 0 910 4832 38 15 35 0 irqbalance
Jun 24 03:40:48 test-host-name kernel: [4329150.494746] [ 927] 103 927 82342 38 59 254 0 whoopsie
Jun 24 03:40:48 test-host-name kernel: [4329150.494751] [ 996] 0 996 2559 58 10 518 0 dhclient
Jun 24 03:40:48 test-host-name kernel: [4329150.494755] [ 1083] 0 1083 19066 27 42 173 0 sshd
Jun 24 03:40:48 test-host-name kernel: [4329150.494758] [ 1177] 0 1177 21359 85 39 131 0 vmtoolsd
Jun 24 03:40:48 test-host-name kernel: [4329150.494762] [ 1290] 106 1290 22904 43 49 231 0 ntpd
Jun 24 03:40:48 test-host-name kernel: [4329150.494765] [ 1359] 0 1359 207408 245 67 178 0 nscd
Jun 24 03:40:48 test-host-name kernel: [4329150.494769] [ 1478] 0 1478 4979 29 15 42 0 master
Jun 24 03:40:48 test-host-name kernel: [4329150.494773] [ 1491] 108 1491 5428 32 15 44 0 qmgr
Jun 24 03:40:48 test-host-name kernel: [4329150.494776] [ 1619] 0 1619 52142 5651 93 14150 0 splunkd
Jun 24 03:40:48 test-host-name kernel: [4329150.494780] [ 1620] 0 1620 14636 1 23 1735 -1000 splunkd
Jun 24 03:40:48 test-host-name kernel: [4329150.494783] [ 1692] 100 1692 4911 63 12 210 0 nagios3
Jun 24 03:40:48 test-host-name kernel: [4329150.494790] [ 1922] 0 1922 1901 2 9 28 0 getty
Jun 24 03:40:48 test-host-name kernel: [4329150.494793] [ 2893] 100 2893 4424 15 11 105 0 nrpe
Jun 24 03:40:48 test-host-name kernel: [4329150.494797] [ 3021] 999 3021 46156 20287 90 8730 0 gmond
Jun 24 03:40:48 test-host-name kernel: [4329150.494803] [ 9911] 0 9911 4870 0 14 55 0 upstart-udev-br
Jun 24 03:40:48 test-host-name kernel: [4329150.494812] [ 9914] 0 9914 10346 1 24 100 -1000 systemd-udevd
Jun 24 03:40:48 test-host-name kernel: [4329150.494818] [10000] 0 10000 8809 36 22 45 0 systemd-logind
Jun 24 03:40:48 test-host-name kernel: [4329150.494823] [10797] 0 10797 45843 272 62 1458 0 apache2
Jun 24 03:40:48 test-host-name kernel: [4329150.494828] [ 9751] 33 9751 45865 360 57 1396 0 apache2
Jun 24 03:40:48 test-host-name kernel: [4329150.494831] [ 9752] 33 9752 45956 390 58 1412 0 apache2
Jun 24 03:40:48 test-host-name kernel: [4329150.494836] [ 9753] 33 9753 45865 361 57 1395 0 apache2
Jun 24 03:40:48 test-host-name kernel: [4329150.494840] [ 9754] 33 9754 45867 363 57 1395 0 apache2
Jun 24 03:40:48 test-host-name kernel: [4329150.494845] [ 9755] 33 9755 45865 362 57 1394 0 apache2
Jun 24 03:40:48 test-host-name kernel: [4329150.494851] [ 2545] 100 2545 4460 0 13 128 0 nrpe
Jun 24 03:40:48 test-host-name kernel: [4329150.494855] [ 3688] 108 3688 8582 41 22 97 0 tlsmgr
Jun 24 03:40:48 test-host-name kernel: [4329150.494859] [ 3708] 33 3708 45956 393 58 1409 0 apache2
Jun 24 03:40:48 test-host-name kernel: [4329150.494865] [ 3729] 33 3729 45861 361 57 1391 0 apache2
Jun 24 03:40:48 test-host-name kernel: [4329150.494869] [13954] 107 13954 2084176 1418568 3304 211435 0 java
Jun 24 03:40:48 test-host-name kernel: [4329150.494874] [30745] 108 30745 5389 50 15 0 0 pickup
Jun 24 03:40:48 test-host-name kernel: [4329150.494878] Out of memory: Kill process 13954 (java) score 713 or sacrifice child
Jun 24 03:40:48 test-host-name kernel: [4329150.509555] Killed process 13954 (java) total-vm:8336704kB, anon-rss:5674272kB, file-rss:0kB
从日志中可以清楚地看出，由于内存不足，内核已终止进程。但是我想知道如何调试这个问题，有什么工具可以帮助我找出问题所在吗？
最佳答案
你为 Tomcat 预留了多少内存？ 看看这个页面:http://www.mkyong.com/tomcat/tomcat-javalangoutofmemoryerror-permgen-space/
关于java - 如何知道为什么我的tomcat被内核杀死，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/38013101/