java - 即使内存被回收，并发模式也会失败

Question

这是我的应用程序的 VM OPS

-Xms7500m
-Xmx7500m
-Xmn4g
-XX:MaxPermSize=192m
-XX:TargetSurvivorRatio=80
-XX:+AggressiveOpts
-XX:+UseFastAccessorMethods
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:ConcGCThreads=6
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCCause
-verbose:gc
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintTenuringDistribution
-XX:+HeapDumpOnOutOfMemoryError

5天内出现3次并发模式异常。应用程序仍在运行文件并且内存也可用。

2020-04-16T18:57:47.575-0400: 100509.755: [CMS-concurrent-abortable-preclean-start]
2020-04-16T18:57:51.701-0400: 100513.881: [GC (Allocation Failure)2020-04-16T18:57:51.702-0400: 100513.881: [ParNew (promotion failed)
Desired survivor size 343565920 bytes, new threshold 6 (max 6)
- age   1:   91240280 bytes,   91240280 total
- age   2:   52703592 bytes,  143943872 total
- age   3:   26770336 bytes,  170714208 total
- age   4:   29495504 bytes,  200209712 total
- age   5:    9595480 bytes,  209805192 total
- age   6:   24205808 bytes,  234011000 total
: 3710667K->3614781K(3774912K), 1.1252750 secs]2020-04-16T18:57:52.827-0400: 100515.007: [CMS CMS: abort preclean due to time 2020-04-16T18:57:52.894-0400: 100515.074: [CMS-concurrent-abortable-preclean: 4.174/5.319 secs] [Times: user=14.39 sys=0.29, real=5.32 secs]
(concurrent mode failure): 3227087K->1442206K(3485696K), 8.0583310 secs] 6926256K->1442206K(7260608K), [CMS Perm : 105501K->105321K(176020K)], 9.1844190 secs] [Times: user=9.41 sys=0.01, real=9.18 secs]
--
2020-04-19T18:05:09.581-0400: 356551.761: [CMS-concurrent-abortable-preclean-start]
2020-04-19T18:05:11.759-0400: 356553.939: [GC (Allocation Failure)2020-04-19T18:05:11.760-0400: 356553.939: [ParNew (promotion failed)
Desired survivor size 343565920 bytes, new threshold 6 (max 6)
- age   1:   95822816 bytes,   95822816 total
- age   2:   24589528 bytes,  120412344 total
- age   3:   28175272 bytes,  148587616 total
- age   4:   24536120 bytes,  173123736 total
- age   5:   23041104 bytes,  196164840 total
- age   6:   12194152 bytes,  208358992 total
: 3670487K->3606232K(3774912K), 0.9360540 secs]2020-04-19T18:05:12.696-0400: 356554.875: [CMS2020-04-19T18:05:12.758-0400: 356554.938: [CMS-concurrent-abortable-preclean: 2.224/3.177 secs] [Times: user=10.62 sys=0.17, real=3.18 secs]
(concurrent mode failure): 3233090K->1492098K(3485696K), 7.9204130 secs] 6896158K->1492098K(7260608K), [CMS Perm : 105666K->105467K(176212K)], 8.8569600 secs] [Times: user=9.08 sys=0.01, real=8.86 secs]
--
2020-04-22T19:07:04.975-0400: 619467.155: Total time for which application threads were stopped: 0.0047280 seconds
2020-04-22T19:07:07.174-0400: 619469.354: [GC (Allocation Failure)2020-04-22T19:07:07.174-0400: 619469.354: [ParNew (promotion failed)
Desired survivor size 343565920 bytes, new threshold 6 (max 6)
- age   1:   98089096 bytes,   98089096 total
- age   2:   31239384 bytes,  129328480 total
- age   3:   29372368 bytes,  158700848 total
- age   4:   27791800 bytes,  186492648 total
- age   5:   19365904 bytes,  205858552 total
- age   6:   35928016 bytes,  241786568 total
: 3643909K->3678567K(3774912K), 0.9460110 secs]2020-04-22T19:07:08.121-0400: 619470.300: [CMS2020-04-22T19:07:08.234-0400: 619470.413: [CMS-concurrent-abortable-preclean: 2.612/3.582 secs] [Times: user=12.78 sys=0.18, real=3.58 secs]
(concurrent mode failure): 3230258K->1503933K(3485696K), 8.8236640 secs] 6862317K->1503933K(7260608K), [CMS Perm : 105907K->105647K(176596K)], 9.7702040 secs] [Times: user=10.06 sys=0.00, real=9.77 secs]

应用在过去一周内没有发生 FullGC

2020-04-15T15:03:29.193-0400: 51.372: [Full GC (Permanent Generation Full)2020-04-15T15:03:29.193-0400: 51.373: [CMS: 771051K->1044286K(3485696K), 4.2744340 secs] 1915012K->1044286K(7260608K), [CMS Perm : 101698K->101581K(102128K)], 4.2749400 secs] [Times: user=4.18 sys=0.11, real=4.28 secs]

为什么会出现并发模式失败异常

1. 不频繁进行 Full GC

2. GC 后内存被回收

   3227087K->1442206K(3485696K), 8.0583310 secs] 6926256K->1442206K(7260608K)
   3233090K->1492098K(3485696K), 7.9204130 secs] 6896158K->1492098K(7260608K)
   3230258K->1503933K(3485696K), 8.8236640 secs] 6862317K->1503933K(7260608K)
   

节点配置:8核CPU、12 GB RAM RHEL VM、JDK 1.7.0_45

Answer 1

来自第 8 章 Concurrent Mark Sweep (CMS) Collector Java 平台标准版 HotSpot 虚拟机垃圾收集调整指南:

Concurrent Mode Failure

The CMS collector uses one or more garbage collector threads that run simultaneously with the application threads with the goal of completing the collection of the tenured generation before it becomes full. As described previously, in normal operation, the CMS collector does most of its tracing and sweeping work with the application threads still running, so only brief pauses are seen by the application threads. However, if the CMS collector is unable to finish reclaiming the unreachable objects before the tenured generation fills up, or if an allocation cannot be satisfied with the available free space blocks in the tenured generation, then the application is paused and the collection is completed with all the application threads stopped. The inability to complete a collection concurrently is referred to as concurrent mode failure and indicates the need to adjust the CMS collector parameters. If a concurrent collection is interrupted by an explicit garbage collection (System.gc()) or for a garbage collection needed to provide information for diagnostic tools, then a concurrent mode interruption is reported.

这意味着，当时您的应用程序生成垃圾的速度比收集器线程回收它们的速度快，并且/或使用了太多 CPU，收集器线程没有时间完成作业。