java - 高 cpu/gc Activity 随后崩溃

Question

我正在运行具有高请求/秒速率的 JAVA Jetty Web 服务器(带有 Jetty Maven 插件)。

启动一段时间后，该应用程序几乎使用了 100% 的 CPU。通过一些研究，我们发现 GC 在崩溃之前多次运行 full gc。

Maven 和 Java 选项:

MAVEN_OPTS="-XX:PermSize=128m -XX:MaxPermSize=512m -Xmx6000m -javaagent:/home/apalabrados/.m2/repository/org/springframework/spring-instrument/3.1.1.RELEASE/spring-instrument-3.1.1.RELEASE.jar -javaagent:/opt/newrelic/newrelic.jar -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=54.226.189.175 -Xloggc:/mnt/gc.log"

Maven-版本:

Apache Maven 3.0.4 (r1232337; 2012-01-17 03:44:56-0500)
Maven home: /opt/maven
Java version: 1.7.0_60, vendor: Oracle Corporation
Java home: /opt/java/jdk1.7.0_60/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.16.33-xenu", arch: "amd64", family: "unix"

这是 jvisualvm 输出:

不要介意下午 5:17 左右的暂停。我们在那里运行了一个堆转储。在没有运行它的情况下，这种情况发生了很多次。

我也收集了 GC 日志，但没有得到任何重要的信息，除了我们有大量的对象创建和释放之外。

有人经历过类似的情况并可以指导我找到一个想法或工具来分析这个问题吗？有没有办法通过GC扫描分析来检测内存泄漏？

这是 gc.log:

1.499: [GC 61440K->7644K(235520K), 0.0120470 secs]
2.699: [GC 69084K->10335K(235520K), 0.0134050 secs]
3.279: [GC 71775K->9751K(235520K), 0.0124680 secs]
3.831: [GC 71191K->12271K(296960K), 0.0244350 secs]
5.659: [GC 135151K->17751K(296960K), 0.0301720 secs]
6.428: [GC 140631K->19051K(424960K), 0.0258600 secs]
6.792: [GC 264811K->18709K(425472K), 0.0202690 secs]
7.073: [GC 264469K->18709K(670720K), 0.0092640 secs]
7.757: [GC 509205K->18711K(671744K), 0.0064060 secs]
8.264: [GC 509207K->18775K(966144K), 0.0013700 secs]
9.385: [GC 803159K->18775K(966656K), 0.0021650 secs]
10.470: [GC 803159K->18775K(1438208K), 0.0011310 secs]
12.365: [GC 1274199K->18839K(1360384K), 0.0011500 secs]
13.577: [GC 1214871K->18847K(1303552K), 0.0012290 secs]
14.718: [GC 1158047K->18855K(1267712K), 0.0011240 secs]
15.835: [GC 1104295K->18863K(1198592K), 0.0053350 secs]
16.935: [GC 1053103K->18839K(1166848K), 0.0014530 secs]
17.949: [GC 1004439K->18975K(1103872K), 0.0051870 secs]
18.924: [GC 958495K->18935K(1075200K), 0.0011990 secs]
19.878: [GC 914423K->19000K(1018368K), 0.0046680 secs]
20.968: [GC 873016K->19000K(992768K), 0.0010440 secs]
22.050: [GC 833592K->19000K(941568K), 0.1194060 secs]
23.053: [GC 796216K->19064K(918528K), 0.0014730 secs]
23.813: [GC 760440K->19072K(871936K), 0.0040810 secs]
24.540: [GC 726656K->19048K(850944K), 0.0014350 secs]
25.224: [GC 694376K->19088K(808960K), 0.0040980 secs]
25.866: [GC 663696K->19088K(789504K), 0.0014640 secs]
26.481: [GC 634512K->19152K(752128K), 0.0036950 secs]
27.218: [GC 606928K->19128K(734720K), 0.0014220 secs]
27.888: [GC 580792K->19160K(700928K), 0.0035020 secs]
28.440: [GC 555736K->19256K(684544K), 0.0015560 secs]
31.329: [GC 781156K->314136K(949760K), 0.0042560 secs]
37.385: [GC 804120K->549570K(1289216K), 0.2488560 secs]
49.073: [GC 1162434K->594663K(1261568K), 0.0662530 secs]
49.140: [Full GC 594663K->82634K(882688K), 0.3667660 secs]
53.656: [GC 667850K->117231K(1000448K), 0.0428860 secs]
59.351: [GC 792559K->143832K(1128448K), 0.1040000 secs]
61.878: [GC 929752K->643702K(1555456K), 0.6804850 secs]
62.558: [Full GC 643702K->617207K(1963520K), 2.8812430 secs]
67.391: [GC 1527543K->1000573K(2088960K), 0.7067240 secs]
68.097: [Full GC 1000573K->991784K(2585088K), 3.1011780 secs]
72.553: [GC 1841192K->1314882K(2701312K), 0.6809580 secs]
74.965: [GC 2136253K->1844959K(2574848K), 0.9235580 secs]
75.889: [Full GC 1844959K->1842339K(3190272K), 6.0295190 secs]
82.983: [GC 2517667K->2044889K(3423744K), 0.9600750 secs]
83.943: [Full GC 2044889K->1831781K(4016128K), 4.2142270 secs]
89.234: [GC 2505682K->2185552K(4028928K), 0.4127690 secs]
90.782: [GC 2891247K->2375033K(3916800K), 0.5920890 secs]
92.957: [GC 3086201K->2564333K(4027392K), 0.7920020 secs]
95.896: [GC 3247341K->2626200K(4027392K), 0.5745350 secs]
96.471: [Full GC 2626200K->2619897K(5012992K), 5.5746770 secs]
103.773: [GC 3302905K->2681641K(5012992K), 0.1234280 secs]
168.549: [GC 3364649K->2676119K(5012992K), 0.1320450 secs]
170.700: [GC 3371999K->3051400K(5012992K), 0.1319660 secs]
172.516: [GC 3734410K->2682189K(5012992K), 0.1214520 secs]
176.063: [GC 3365197K->2671799K(5012992K), 0.1606770 secs]
179.177: [GC 3354807K->2671123K(5012992K), 0.1226220 secs]
181.998: [GC 3354131K->2670544K(4976128K), 0.1093350 secs]
184.752: [GC 3353552K->2671647K(4335104K), 0.1159500 secs]
187.379: [GC 3354655K->2669489K(4937216K), 0.1204930 secs]
189.888: [GC 3369905K->2670348K(4351488K), 0.0916450 secs]
192.076: [GC 3370764K->2671982K(4896256K), 0.0905400 secs]
194.073: [GC 3388270K->2671739K(4368384K), 0.1035390 secs]
196.116: [GC 3388027K->2671760K(4854272K), 0.0939740 secs]
197.905: [GC 3402896K->2674545K(4386304K), 0.0913640 secs]
199.688: [GC 3405681K->2677264K(4814848K), 0.0949850 secs]
201.516: [GC 3423760K->2696736K(4423680K), 0.1155460 secs]
203.142: [GC 3443232K->2711049K(4768768K), 0.1137520 secs]
205.239: [GC 3474953K->2703962K(4448256K), 0.1208140 secs]
207.405: [GC 3467866K->2707451K(4736000K), 0.1211140 secs]
209.625: [GC 3488763K->2705569K(4467200K), 0.1238950 secs]
212.012: [GC 3486881K->2702818K(4712960K), 0.1044980 secs]
214.086: [GC 3501026K->2702421K(4480512K), 0.0865790 secs]
216.401: [GC 3500629K->2706221K(4694016K), 0.1068410 secs]
218.647: [GC 3520813K->2702911K(4712960K), 0.1104230 secs]
220.946: [GC 3517503K->2704774K(4682240K), 0.1222490 secs]
222.996: [GC 3536774K->2705598K(4697600K), 0.1077060 secs]
225.043: [GC 3537598K->2705634K(4675072K), 0.1222020 secs]
227.271: [GC 3556066K->2702994K(4687872K), 0.1027970 secs]
229.888: [GC 3553426K->2701963K(4671488K), 0.1179580 secs]
232.156: [GC 3570315K->2703940K(4682752K), 0.1054530 secs]
234.472: [GC 3572292K->2702254K(4674048K), 0.1011400 secs]
236.784: [GC 3588526K->2703318K(4682240K), 0.1181500 secs]
238.888: [GC 3589590K->2714847K(4681216K), 0.1103460 secs]
240.745: [GC 3620063K->2738332K(4688384K), 0.1200060 secs]
243.287: [GC 3643548K->2738131K(4682240K), 0.1030970 secs]
245.779: [GC 3663315K->2746074K(4691456K), 0.1116630 secs]
248.299: [GC 3671258K->2737044K(4690432K), 0.1028660 secs]
250.721: [GC 3681172K->2736814K(4698624K), 0.1163070 secs]
253.395: [GC 3680942K->2741374K(4700672K), 0.1077050 secs]
257.305: [GC 3704446K->2744574K(4705792K), 0.1154280 secs]
259.993: [GC 3707646K->2747490K(4715008K), 0.0973580 secs]
262.478: [GC 3727970K->2746337K(4717568K), 0.1174640 secs]
265.214: [GC 3726817K->2747432K(4729344K), 0.1168990 secs]
268.018: [GC 3746344K->2748090K(4731392K), 0.0966240 secs]
271.150: [GC 3747002K->2747048K(4741120K), 0.1082590 secs]
273.614: [GC 3763368K->2759423K(4744192K), 0.1082620 secs]
276.447: [GC 3775743K->2759649K(4763648K), 0.1237910 secs]
279.371: [GC 3792865K->2756841K(4765184K), 0.1113540 secs]
281.532: [GC 3790057K->2757896K(4782080K), 0.1078550 secs]
284.264: [GC 3810568K->2757376K(4781568K), 0.1065630 secs]
287.181: [GC 3810048K->2759906K(4795392K), 0.0879670 secs]
289.789: [GC 3831010K->2759543K(4797952K), 0.0994050 secs]
292.294: [GC 3830647K->2772912K(4809728K), 0.1130040 secs]
294.815: [GC 3863472K->2806710K(4812800K), 0.1195040 secs]
297.782: [GC 3897270K->2804618K(4836864K), 0.1245710 secs]
300.448: [GC 3912074K->2807630K(4840448K), 0.1211280 secs]
303.155: [GC 3915086K->2807308K(4864000K), 0.1191050 secs]
306.093: [GC 3934732K->2809680K(4865536K), 0.1268630 secs]
308.853: [GC 3937104K->2818580K(4885504K), 0.1246500 secs]
312.046: [GC 3963412K->2815443K(4890112K), 0.1173920 secs]
314.725: [GC 3960275K->2826572K(4911104K), 0.1553190 secs]
318.089: [GC 3990348K->2826604K(4913664K), 0.1324180 secs]
321.033: [GC 3990380K->2827701K(4936704K), 0.1363310 secs]
324.272: [GC 4012981K->2825759K(4938240K), 0.1345190 secs]
327.234: [GC 4011039K->2831580K(4959744K), 0.1375630 secs]
330.314: [GC 4038364K->2827398K(4961280K), 0.1283290 secs]
333.229: [GC 4034182K->2827799K(4981760K), 0.1313840 secs]
336.596: [GC 4058647K->2830961K(4982784K), 0.1273130 secs]
339.917: [GC 4061809K->2829995K(5005312K), 0.1347070 secs]
343.168: [GC 4083883K->2829372K(5005824K), 0.1173740 secs]
346.708: [GC 4083260K->2830356K(5026816K), 0.1170540 secs]
349.755: [GC 4106260K->2830115K(5027840K), 0.1122110 secs]
352.978: [GC 4106019K->2829393K(5045760K), 0.1214530 secs]
356.502: [GC 4127313K->2834253K(5047296K), 0.1205870 secs]
359.867: [GC 4132173K->2835471K(5065728K), 0.0889700 secs]
363.525: [GC 4153871K->2840666K(5067776K), 0.1161790 secs]
367.442: [GC 4159066K->2842260K(5082112K), 0.0987980 secs]
371.062: [GC 4180116K->2847297K(5084672K), 0.1086770 secs]
374.661: [GC 4185153K->2857290K(5093376K), 0.1313160 secs]
378.565: [GC 4215114K->2856874K(5097472K), 0.1254220 secs]
382.094: [GC 4214698K->2857452K(5105664K), 0.1202040 secs]
385.370: [GC 4235756K->2881525K(5109760K), 0.1226390 secs]
388.618: [GC 4259829K->2880106K(5132800K), 0.1269360 secs]
392.075: [GC 4279402K->2879101K(5133312K), 0.1265580 secs]
395.698: [GC 4278397K->2879578K(5154304K), 0.1202890 secs]
398.427: [GC 4301402K->2906754K(5154816K), 0.1270330 secs]
401.953: [GC 4328578K->2907187K(5181440K), 0.1377130 secs]
405.726: [GC 4346419K->2906453K(5188608K), 0.1348810 secs]
409.319: [GC 4345685K->2901761K(5216768K), 0.1356850 secs]
412.836: [GC 4363521K->2914802K(5218816K), 0.1330720 secs]
416.075: [GC 4376562K->2928451K(5240832K), 0.1488830 secs]
420.181: [GC 4407107K->2924474K(5249024K), 0.1415160 secs]
424.048: [GC 4403130K->2926226K(5277184K), 0.1395830 secs]
427.646: [GC 4430482K->2928018K(5277184K), 0.1276150 secs]
431.113: [GC 4432274K->2925485K(5291008K), 0.1198850 secs]
434.863: [GC 4454829K->2940590K(5295616K), 0.1382240 secs]
438.851: [GC 4469934K->2938781K(5311488K), 0.1295270 secs]
442.884: [GC 4492701K->2937566K(5317120K), 0.1440960 secs]
446.898: [GC 4491486K->2944314K(5330432K), 0.1364070 secs]
451.133: [GC 4523322K->2937894K(5334528K), 0.1053120 secs]
455.043: [GC 4516902K->2946759K(5355520K), 0.1445050 secs]
459.010: [GC 4549831K->2947447K(5355008K), 0.1048870 secs]
462.689: [GC 4550519K->2946671K(5374976K), 0.1219330 secs]
466.503: [GC 4573295K->2963393K(5376000K), 0.1270760 secs]
470.927: [GC 4590017K->2959873K(5392384K), 0.1229860 secs]
475.170: [GC 4610049K->2959712K(5394432K), 0.1192640 secs]
479.092: [GC 4609888K->2969327K(5413888K), 0.1104220 secs]
482.815: [GC 4642543K->2985450K(5415936K), 0.1124460 secs]
486.748: [GC 4658666K->2982265K(5432832K), 0.1302470 secs]
490.046: [GC 4679033K->3022858K(5434880K), 0.1427400 secs]
494.466: [GC 4719626K->3022955K(5471232K), 0.1644750 secs]
498.716: [GC 4743787K->3029034K(5473280K), 0.1307170 secs]
502.135: [GC 4749866K->3078764K(5481984K), 0.1810790 secs]
505.446: [GC 4804716K->3096042K(5503488K), 0.1507150 secs]
509.649: [GC 4821994K->3097323K(5529600K), 0.1586240 secs]
514.209: [GC 4836075K->3096819K(5506048K), 0.1399260 secs]
517.525: [GC 4835571K->3140631K(5492224K), 0.1750810 secs]
522.107: [GC 4824087K->3134775K(5474304K), 0.1708960 secs]
526.575: [GC 4818231K->3133871K(5489664K), 0.1618120 secs]
531.061: [GC 4784559K->3132405K(5438976K), 0.1456150 secs]
534.842: [GC 4783093K->3160326K(5466112K), 0.1893020 secs]
538.979: [GC 4776198K->3161687K(5424640K), 0.1588550 secs]
543.380: [GC 4777559K->3155091K(5474304K), 0.1695350 secs]
547.073: [GC 4758675K->3184611K(5421056K), 0.1952130 secs]
551.148: [GC 4788195K->3187849K(5451776K), 0.1654520 secs]
554.777: [GC 4762761K->3181217K(5388288K), 0.1827070 secs]
559.019: [GC 4756129K->3181157K(5456896K), 0.1886390 secs]
563.046: [GC 4751461K->3185223K(5350912K), 0.1745280 secs]
565.998: [GC 4755527K->3252200K(5443584K), 0.2152420 secs]
569.709: [GC 4819432K->3267572K(5446656K), 0.1809990 secs]
573.586: [GC 4834804K->3267739K(5449216K), 0.1867050 secs]
577.454: [GC 4831387K->3265648K(5349376K), 0.1869730 secs]
581.502: [GC 4829296K->3276802K(5456896K), 0.1580800 secs]
584.815: [GC 4852226K->3281238K(5449216K), 0.1770960 secs]
588.950: [GC 4856662K->3279733K(5461504K), 0.1635150 secs]
592.871: [GC 4873589K->3285744K(5455360K), 0.1722840 secs]
596.880: [GC 4879600K->3290580K(5475328K), 0.1827670 secs]
601.325: [GC 4909012K->3288510K(5470720K), 0.1264210 secs]
604.174: [GC 4906942K->3343347K(5478912K), 0.1988830 secs]
608.151: [GC 4974067K->3345739K(5481984K), 0.1856410 secs]
612.292: [GC 4976459K->3348003K(5487104K), 0.1764590 secs]
616.020: [GC 4982307K->3363644K(5465600K), 0.1915960 secs]
620.204: [GC 4997948K->3356767K(5478912K), 0.1866870 secs]
624.531: [GC 4974687K->3353409K(5381632K), 0.2094420 secs]
628.655: [GC 4971329K->3365057K(5479936K), 0.1617780 secs]
632.223: [GC 4981953K->3380315K(5392896K), 0.1734390 secs]
636.319: [GC 4997211K->3394047K(5482496K), 0.1676130 secs]
640.879: [GC 5017087K->3394525K(5476352K), 0.1620290 secs]
644.995: [GC 5017565K->3397318K(5481472K), 0.1679770 secs]
649.320: [GC 5029574K->3400067K(5479424K), 0.1512770 secs]
653.463: [GC 5032323K->3406275K(5492736K), 0.1787830 secs]
657.074: [GC 5055939K->3424181K(5491712K), 0.1814630 secs]
661.627: [GC 5073845K->3419096K(5496832K), 0.1614760 secs]
666.063: [GC 5081048K->3419336K(5494784K), 0.1786270 secs]
670.363: [GC 5081288K->3423488K(5502464K), 0.1395200 secs]
674.741: [GC 5089536K->3425836K(5402112K), 0.1573880 secs]
679.237: [GC 5091884K->3423987K(5499904K), 0.1534250 secs]
683.492: [GC 5084915K->3425348K(5501952K), 0.1477920 secs]
687.255: [GC 5086276K->3435330K(5496832K), 0.1494340 secs]
690.765: [GC 5096258K->3487387K(5499392K), 0.1873510 secs]
693.980: [GC 5148315K->3538570K(5483520K), 0.1974280 secs]
698.362: [GC 5186186K->3545357K(5452288K), 0.1753090 secs]
702.342: [GC 5192973K->3540763K(5483008K), 0.1718210 secs]
706.158: [GC 5169947K->3539836K(5486080K), 0.1676040 secs]
710.498: [GC 5169020K->3533045K(5476352K), 0.1508310 secs]
714.464: [GC 5163765K->3538685K(5481984K), 0.1603150 secs]
718.520: [GC 5169405K->3540570K(5478400K), 0.1560510 secs]
721.950: [GC 5185626K->3542957K(5485568K), 0.1610350 secs]
725.698: [GC 5188013K->3544356K(5490176K), 0.1614740 secs]
729.577: [GC 5212452K->3545439K(5496832K), 0.1571760 secs]
733.983: [GC 5213535K->3546135K(5502976K), 0.1411260 secs]
738.289: [GC 5236247K->3546909K(5509632K), 0.1606130 secs]
742.542: [GC 5237021K->3551334K(5513728K), 0.1477740 secs]
746.287: [GC 5259366K->3572688K(5519872K), 0.1700300 secs]
750.480: [GC 5280720K->3576025K(5528576K), 0.1558250 secs]
754.954: [GC 5290201K->3578800K(5423616K), 0.1813190 secs]
759.506: [GC 5292976K->3577713K(5510144K), 0.1082080 secs]
763.631: [GC 5266289K->3580202K(5386752K), 0.1421280 secs]
768.308: [GC 5268778K->3580052K(5508608K), 0.1076640 secs]
771.933: [GC 5257364K->3616569K(5411328K), 0.1415520 secs]
775.366: [GC 5293881K->3653385K(5507072K), 0.1681310 secs]
779.364: [GC 5333257K->3661018K(5506048K), 0.1551540 secs]
783.775: [GC 5340890K->3660401K(5516288K), 0.1548570 secs]
787.665: [GC 5350001K->3653284K(5446656K), 0.1538160 secs]
791.435: [GC 5342884K->3676352K(5507072K), 0.1529650 secs]
796.029: [GC 5362880K->3662743K(5510144K), 0.1563520 secs]
800.558: [GC 5349271K->3662844K(5513728K), 0.1710580 secs]
804.906: [GC 5362172K->3663230K(5516288K), 0.1482320 secs]
809.315: [GC 5362558K->3662776K(5523456K), 0.1484450 secs]
814.172: [GC 5376952K->3669442K(5523456K), 0.1614000 secs]
817.854: [GC 5383618K->3670707K(5531136K), 0.1444250 secs]
822.157: [GC 5397683K->3668915K(5530112K), 0.1498340 secs]
826.481: [GC 5395891K->3673408K(5536768K), 0.1441890 secs]
830.836: [GC 5412160K->3674472K(5536256K), 0.1558090 secs]
834.991: [GC 5413224K->3673214K(5540352K), 0.1691030 secs]
839.413: [GC 5419134K->3684030K(5541888K), 0.1322270 secs]
839.545: [Full GC 3684030K->3479096K(5984256K), 5.1058640 secs]
848.759: [GC 5225016K->3485627K(5966848K), 0.1112250 secs]
853.141: [GC 5210043K->3488417K(5824000K), 0.1014540 secs]
856.887: [GC 5212833K->3501762K(5969408K), 0.0990740 secs]
861.232: [GC 5213890K->3498619K(5969920K), 0.1064590 secs]
865.507: [GC 5210747K->3514189K(5960704K), 0.1138140 secs]
869.542: [GC 5226829K->3510076K(5965312K), 0.1075920 secs]
873.703: [GC 5222716K->3512187K(5964288K), 0.0982860 secs]
877.902: [GC 5238651K->3517052K(5969920K), 0.1203500 secs]
882.443: [GC 5243516K->3517244K(5972480K), 0.1191820 secs]
886.780: [GC 5264188K->3541051K(5978112K), 0.1290750 secs]
890.731: [GC 5287995K->3572478K(5989888K), 0.1464980 secs]
894.936: [GC 5336830K->3573200K(5948416K), 0.1350820 secs]
899.107: [GC 5337552K->3579456K(5980160K), 0.1477950 secs]
903.673: [GC 5323328K->3568973K(5923840K), 0.1274530 secs]
908.371: [GC 5312845K->3568171K(5986304K), 0.1311430 secs]
912.534: [GC 5312043K->3582713K(5933056K), 0.1330360 secs]
915.725: [GC 5326585K->3639941K(5938688K), 0.1866970 secs]
919.691: [GC 5334661K->3667535K(5954560K), 0.1962910 secs]
923.795: [GC 5362255K->3660478K(5890048K), 0.1748390 secs]
927.835: [GC 5277374K->3658735K(5855232K), 0.1681820 secs]
931.503: [GC 5275631K->3664424K(5901312K), 0.1634220 secs]
935.357: [GC 5253672K->3655628K(5824000K), 0.1780940 secs]
938.900: [GC 5244876K->3661179K(5906944K), 0.1687800 secs]
942.720: [GC 5247355K->3661963K(5820416K), 0.1739330 secs]
946.781: [GC 5248139K->3661499K(5913088K), 0.1887860 secs]
950.974: [GC 5254331K->3661484K(5780480K), 0.1580460 secs]
954.912: [GC 5254316K->3663208K(5906944K), 0.1338090 secs]
958.525: [GC 5263720K->3663377K(5900288K), 0.1261580 secs]
962.429: [GC 5263889K->3667408K(5921280K), 0.1433710 secs]
965.882: [GC 5295568K->3706694K(5915648K), 0.1692140 secs]
969.484: [GC 5334854K->3710133K(5937664K), 0.1629000 secs]
973.843: [GC 5359797K->3708370K(5935616K), 0.1446010 secs]
977.919: [GC 5358034K->3711220K(5937152K), 0.1383530 secs]
982.010: [GC 5364980K->3701390K(5937152K), 0.1418670 secs]
986.114: [GC 5355150K->3700150K(5943296K), 0.1095810 secs]
989.285: [GC 5361078K->3733760K(5832704K), 0.1438740 secs]
993.530: [GC 5394688K->3755676K(5944320K), 0.1499240 secs]
997.651: [GC 5424284K->3761001K(5938176K), 0.1528680 secs]
1002.093: [GC 5429609K->3761553K(5950976K), 0.1407370 secs]
1005.544: [GC 5449617K->3783002K(5945344K), 0.1458060 secs]
1009.786: [GC 5471066K->3775461K(5968384K), 0.1531330 secs]
1014.041: [GC 5484005K->3776869K(5914624K), 0.1569560 secs]
1018.223: [GC 5485413K->3775740K(5968384K), 0.1536780 secs]
1022.691: [GC 5486844K->3779760K(5966336K), 0.1502870 secs]
1026.966: [GC 5490864K->3777283K(5969408K), 0.1608990 secs]
1030.877: [GC 5494019K->3785851K(5967872K), 0.1295130 secs]
1035.407: [GC 5502587K->3784620K(5973504K), 0.1413430 secs]
1039.677: [GC 5511596K->3785988K(5971968K), 0.1356230 secs]
1044.069: [GC 5512964K->3790855K(5978624K), 0.1259940 secs]
1047.575: [GC 5531143K->3825864K(5977600K), 0.1515640 secs]
1052.126: [GC 5566152K->3820289K(5985792K), 0.1656510 secs]
1056.439: [GC 5566721K->3814956K(5984768K), 0.1364480 secs]
1060.644: [GC 5561388K->3818750K(5988864K), 0.1349440 secs]
1064.670: [GC 5569278K->3820566K(5988352K), 0.1580760 secs]
1068.918: [GC 5571094K->3822254K(5985792K), 0.1277440 secs]
1073.599: [GC 5571246K->3822567K(5886464K), 0.1187910 secs]
1077.878: [GC 5571559K->3823792K(5988352K), 0.1188130 secs]
1082.448: [GC 5572272K->3823184K(5885952K), 0.1265090 secs]
1086.745: [GC 5571664K->3827026K(5988864K), 0.1165090 secs]
1090.919: [GC 5578066K->3830794K(5895680K), 0.1149620 secs]
1094.519: [GC 5581834K->3846640K(5990912K), 0.1414780 secs]
1097.585: [GC 5606384K->3903071K(5985792K), 0.1467760 secs]
1101.869: [GC 5662815K->3903986K(5972992K), 0.1605140 secs]
1106.450: [GC 5639666K->3903723K(5940736K), 0.1527220 secs]
1110.362: [GC 5639403K->3928472K(5968384K), 0.1614570 secs]
1114.435: [GC 5645720K->3935890K(5929984K), 0.1611700 secs]
1118.681: [GC 5653138K->3931392K(5963264K), 0.1382810 secs]
1122.908: [GC 5634304K->3910648K(5885952K), 0.1427910 secs]
1127.102: [GC 5613560K->3918401K(5964288K), 0.1556450 secs]
1130.866: [GC 5627969K->3944438K(5959680K), 0.1628050 secs]
1135.252: [GC 5654006K->3952062K(5960704K), 0.1442020 secs]
1139.189: [GC 5658046K->3951277K(5926912K), 0.1607300 secs]
1139.752: [GC 4184078K->3950908K(5957632K), 0.1538450 secs]
1139.906: [Full GC 3950908K->3898572K(5963776K), 5.3527500 secs]
1282.359: [GC 5591244K->3906287K(5796864K), 0.1061990 secs]
1286.347: [GC 5598959K->3910799K(5946368K), 0.1009020 secs]
1290.491: [GC 5573775K->3911727K(5772288K), 0.1112600 secs]
1294.726: [GC 5574703K->3913680K(5949440K), 0.1155420 secs]
1298.817: [GC 5572560K->3917569K(5774336K), 0.1151050 secs]
1302.246: [GC 5576449K->3924991K(5951488K), 0.1052560 secs]
1306.107: [GC 5588991K->3925982K(5946368K), 0.1087310 secs]
1309.462: [GC 5589982K->3953088K(5945856K), 0.1409480 secs]
1313.760: [GC 5633984K->3949086K(5946368K), 0.1432650 secs]
1317.749: [GC 5629982K->3947235K(5966336K), 0.1210100 secs]
1322.148: [GC 5664995K->3958630K(5965824K), 0.1285720 secs]
1326.870: [GC 5676390K->3958323K(5984256K), 0.1339940 secs]
1331.160: [GC 5709363K->3960916K(5982720K), 0.1367300 secs]
1335.567: [GC 5711956K->3974140K(6000640K), 0.1236790 secs]
1339.221: [GC 5749756K->4004699K(6000128K), 0.1578430 secs]
1343.801: [GC 5780315K->4005529K(5992960K), 0.1509180 secs]
1347.885: [GC 5763737K->4013561K(5958656K), 0.1592100 secs]
1352.176: [GC 5771769K->4012942K(5986304K), 0.1495320 secs]
1356.638: [GC 5752718K->4011940K(5937664K), 0.1451850 secs]
1360.708: [GC 5751716K->4012188K(5986816K), 0.1512310 secs]
1365.494: [GC 5746844K->4014059K(5928448K), 0.1419770 secs]
1369.694: [GC 5748715K->4015146K(5989376K), 0.1450500 secs]
1373.648: [GC 5752874K->4013893K(5906432K), 0.1447520 secs]
1378.032: [GC 5751621K->4022270K(5989376K), 0.1225350 secs]
1382.421: [GC 5767678K->4021332K(5983232K), 0.1312980 secs]
1386.569: [GC 5766740K->4025845K(5995008K), 0.1449120 secs]
1390.111: [GC 5788149K->4036550K(5990912K), 0.1379620 secs]
1394.023: [GC 5798854K->4036116K(6006272K), 0.1273070 secs]
1397.732: [GC 5818900K->4049615K(6002176K), 0.1480570 secs]
1401.705: [GC 5832399K->4068217K(6014976K), 0.1587040 secs]
1405.896: [GC 5867897K->4080057K(6012416K), 0.1394640 secs]
1410.089: [GC 5879737K->4075591K(6022144K), 0.1393760 secs]
1414.117: [GC 5888583K->4083024K(6021120K), 0.1322730 secs]
1418.241: [GC 5896016K->4100947K(6026752K), 0.1514540 secs]
1422.422: [GC 5922131K->4096826K(6028288K), 0.1335430 secs]
1426.649: [GC 5918010K->4097608K(6030336K), 0.1356250 secs]
1430.755: [GC 5923912K->4098969K(6031872K), 0.1292980 secs]
1434.792: [GC 5925273K->4101050K(6033920K), 0.1459020 secs]
1439.063: [GC 5930426K->4106264K(6034432K), 0.1440010 secs]
1442.885: [GC 5935640K->4105713K(6031872K), 0.1375000 secs]
1447.144: [GC 5936113K->4106576K(6033408K), 0.1377320 secs]
1451.629: [GC 5936976K->4106264K(6036480K), 0.1541940 secs]
1456.070: [GC 5942296K->4105425K(6036480K), 0.1241690 secs]
1460.307: [GC 5941457K->4114589K(6039040K), 0.1338140 secs]
1465.089: [GC 5954205K->4112849K(5986816K), 0.1317500 secs]
1469.043: [GC 5952465K->4112954K(6030848K), 0.1202860 secs]
1473.189: [GC 5941818K->4116186K(5966848K), 0.1090450 secs]
1477.156: [GC 5945050K->4126626K(6032896K), 0.1193880 secs]
1477.276: [Full GC 4126626K->4078945K(6032896K), 5.2454720 secs]
1485.772: [Full GC 5904737K->4094083K(6032896K), 5.3555330 secs]
1494.693: [Full GC 5919875K->4116448K(6032896K), 10.1368700 secs]
1508.512: [Full GC 5921778K->4120445K(6032896K), 9.1725230 secs]
1521.118: [Full GC 5921459K->4143720K(6032896K), 5.7287810 secs]
1531.071: [Full GC 5921431K->4141759K(6032896K), 5.8122450 secs]
1540.424: [Full GC 5921694K->4167511K(6032896K), 5.4013940 secs]
1549.700: [Full GC 5921548K->4164288K(6032896K), 5.2783960 secs]
1558.860: [Full GC 5921386K->4162930K(6032896K), 6.6543130 secs]
1569.237: [Full GC 5921712K->4163299K(6032896K), 6.0555560 secs]
1579.282: [Full GC 5921590K->4162837K(6032896K), 5.1936220 secs]
1588.510: [Full GC 5921434K->4163549K(6032896K), 6.5699810 secs]
1598.852: [Full GC 5921767K->4167412K(6032896K), 5.8504890 secs]
1608.076: [Full GC 5921603K->4205587K(6032896K), 5.4929970 secs]
1617.220: [Full GC 5921543K->4198405K(6032896K), 6.0725330 secs]
1626.892: [Full GC 5921421K->4192351K(6032896K), 6.0990110 secs]
1636.348: [Full GC 5921334K->4201010K(6032896K), 6.1280120 secs]
1646.380: [Full GC 5921762K->4198287K(6032896K), 5.3571270 secs]
1655.530: [Full GC 5921619K->4205816K(6032896K), 6.2468180 secs]
1662.805: [Full GC 4542963K->1722668K(6032896K), 3.6291470 secs]

Answer 1

我们使用 Eclipse 内存分析器发现了问题。我们让应用程序运行，直到 RAM 几乎耗尽，进行堆转储，并使用 Eclipse 工具找到问题。

事实证明，java-mpns(一个向微软手机发送推送通知的 Java 客户端)使用了一个固定的线程池，其中有一个无界的 LinkedBlockingQueue。尽管它只有 10 个线程来执行任务，但它保持“无限”(Integer.MAX_VALUE) 数量的任务等待。随着时间的推移，这会消耗大量 GC 未释放的 RAM。这就是为什么存在大量 GC Activity 、CPU Activity 以及我们无法跟踪消耗 CPU 的 java 线程的原因。