java - Tomcat 垃圾回收器 : two machines with the same settings behave differently

Question

我有两个具有相同配置的虚拟服务器 (Win 2012R2):64GB 内存和 12 个内核。

每个虚拟服务器都有一个 Tomcat 实例 (8.0.36)，它们通过 SimpleTcpCluster 集群。

关于 GC，我对两者都使用以下内容:

set JAVA_OPTS=-Xms4g 
set JAVA_OPTS=%JAVA_OPTS% -Xmx4g
set JAVA_OPTS=%JAVA_OPTS% -Xss256k
set JAVA_OPTS=%JAVA_OPTS% -XX:+UseG1GC -XX:MaxGCPauseMillis=2000
set JAVA_OPTS=%JAVA_OPTS% -XX:G1HeapRegionSize=8
set JAVA_OPTS=%JAVA_OPTS% -XX:ParallelGCThreads=8 -XX:ConcGCThreads=4 
set JAVA_OPTS=%JAVA_OPTS% -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps

一个 Tomcat 实例性能非常好，我从未发现任何明显的性能下降。我确实看到它经常执行 GC。这是它生成的示例数据:

2016-07-22T09:55:57.164-0400: 36257.947: [GC pause (G1 Evacuation Pause) (young), 0.0408534 secs]
   [Parallel Time: 11.9 ms, GC Workers: 8]
      [GC Worker Start (ms): Min: 36257947.8, Avg: 36257948.0, Max: 36257948.2, Diff: 0.3]
      [Ext Root Scanning (ms): Min: 0.6, Avg: 1.0, Max: 2.4, Diff: 1.8, Sum: 8.0]
      [Update RS (ms): Min: 0.6, Avg: 1.8, Max: 2.1, Diff: 1.4, Sum: 14.4]
         [Processed Buffers: Min: 11, Avg: 16.3, Max: 21, Diff: 10, Sum: 130]
      [Scan RS (ms): Min: 0.7, Avg: 0.9, Max: 1.0, Diff: 0.2, Sum: 7.3]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: 1.2]
      [Object Copy (ms): Min: 7.2, Avg: 7.5, Max: 7.7, Diff: 0.5, Sum: 59.7]
      [Termination (ms): Min: 0.1, Avg: 0.1, Max: 0.1, Diff: 0.0, Sum: 0.9]
         [Termination Attempts: Min: 324, Avg: 360.5, Max: 388, Diff: 64, Sum: 2884]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.0]
      [GC Worker Total (ms): Min: 11.4, Avg: 11.6, Max: 11.7, Diff: 0.3, Sum: 92.6]
      [GC Worker End (ms): Min: 36257959.4, Avg: 36257959.5, Max: 36257959.6, Diff: 0.2]
   [Code Root Fixup: 0.3 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.8 ms]
   [Other: 27.8 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 23.0 ms]
      [Ref Enq: 0.2 ms]
      [Redirty Cards: 0.2 ms]
      [Humongous Register: 0.2 ms]
      [Humongous Reclaim: 0.4 ms]
      [Free CSet: 3.0 ms]
   [Eden: 2406.0M(2406.0M)->0.0B(2401.0M) Survivors: 51.0M->56.0M Heap: 2809.2M(4096.0M)->211.6M(4096.0M)]
 [Times: user=0.06 sys=0.00, real=0.06 secs]

然而，另一个 Tomcat 一直显示缓慢。每次缓慢一段时间后，它都会恢复正常性能。这是它在 GC 中生成的示例数据:

2016-07-22T10:03:22.442-0400: 37959.345: [GC pause (G1 Evacuation Pause) (young), 0.1030285 secs]
   [Parallel Time: 50.4 ms, GC Workers: 8]
      [GC Worker Start (ms): Min: 37959345.4, Avg: 37959345.5, Max: 37959345.6, Diff: 0.2]
      [Ext Root Scanning (ms): Min: 0.4, Avg: 1.2, Max: 5.2, Diff: 4.8, Sum: 9.8]
      [Update RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [Processed Buffers: Min: 0, Avg: 0.0, Max: 0, Diff: 0, Sum: 0]
      [Scan RS (ms): Min: 0.1, Avg: 0.4, Max: 0.5, Diff: 0.4, Sum: 3.2]
      [Code Root Scanning (ms): Min: 0.0, Avg: 7.5, Max: 15.1, Diff: 15.1, Sum: 59.6]
      [Object Copy (ms): Min: 33.6, Avg: 40.6, Max: 48.2, Diff: 14.6, Sum: 325.1]
      [Termination (ms): Min: 0.0, Avg: 0.3, Max: 0.4, Diff: 0.4, Sum: 2.3]
         [Termination Attempts: Min: 1, Avg: 880.9, Max: 1086, Diff: 1085, Sum: 7047]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 0.9]
      [GC Worker Total (ms): Min: 50.0, Avg: 50.1, Max: 50.3, Diff: 0.3, Sum: 400.8]
      [GC Worker End (ms): Min: 37959395.5, Avg: 37959395.6, Max: 37959395.6, Diff: 0.1]
   [Code Root Fixup: 8.7 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.8 ms]
   [Other: 43.1 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 36.5 ms]
      [Ref Enq: 1.2 ms]
      [Redirty Cards: 0.2 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 4.3 ms]
   [Eden: 2300.0M(2300.0M)->0.0B(2346.0M) Survivors: 157.0M->111.0M Heap: 2496.5M(4096.0M)->150.5M(4096.0M)]
 [Times: user=0.42 sys=0.00, real=0.11 secs]

造成这种性能差异的原因可能是什么？我真的不知道从哪里开始。如果您需要有关系统的任何信息，请随时询问。

问候。

更新

这里是更多的系统信息:

• 两个实例运行同一个面向公众的网站(完全相同的代码)。
• 两个实例前面有一个 Kemp 负载平衡器。方法是圆的 罗宾。
• 相同的 Java 版本:1.8.0.92(再次验证)
• 两个实例的负载应该非常相似。
• 网站不在云端。
• 这是一个流量较低的新推出的网站。任何时候平均有 10 - 20 名访客。在这种低流量的情况下，速度仍然很慢。
• 两个 Tomcat 实例位于两个盒子上。
• 在装有慢速 Tomcat 实例的盒子上，我有一个被动 SQL Server 2014 实例，它与快速 Tomcat 实例所在的另一个盒子上的 Activity SQL Server 2014 实例组成集群。
• 两个盒子之间的通信是10G。

Answer 1

当您使用虚拟机时，很可能存在运行在相同硬件容量上的“邻居”。这些邻居可能会窃取您虚拟机的容量。例如，在亚马逊中，如果发生资源分配冲突，首先会优先考虑更大的实例(与 t1.micro 相比，c4.xlarge 将首先获得资源，因此在 t1.micro 处你会看到 steal time 在 top 实用程序)。在我的实践中，当我遇到类似情况时，我会停止并启动缓慢的实例，然后它会移动到另一个硬件。如果这一次，幸运的是有“邻居”，窃取时间就没有了(它适用于亚马逊，但我想它适用于其他管理程序)。您使用的是什么提供商？