java - Tomcat8 使用 AbstractProtocol.pause 随机关闭

Question

1。问题

我在Tomcat中部署了一个webapp，发现它随机关机，时间从2、3小时到2、3天不等。 catalina.out 中的登录是:

26224 2015-06-10 13:59:04.110 {http-nio-8080-exec-3} INFO  com.timediff.controller.user.UserProfileController#getUserHome - /user/profile/home done, curUid: 889
26225 10-Jun-2015 14:15:35.050 INFO [Thread-11] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8080"]
26226 10-Jun-2015 14:15:35.052 INFO [Thread-11] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["ajp-nio-8009"]
26227 10-Jun-2015 14:15:35.053 INFO [Thread-11] org.apache.catalina.core.StandardService.stopInternal Stopping service Catalina
26228 10-Jun-2015 14:15:35.058 INFO [localhost-startStop-2] org.springframework.web.context.support.XmlWebApplicationContext.doClose Closing WebApplicationContext for namespace 'timediff-dispatcher-servlet': startup date [Wed Jun 10 13:38:14 CST 2015]; root of context hierarchy
26229 10-Jun-2015 14:15:35.059 INFO [localhost-startStop-2] org.springframework.context.support.DefaultLifecycleProcessor.stop Stopping beans in phase 2147483647
26230 2015-06-10 14:15:35.061 {localhost-startStop-2} INFO  org.quartz.core.QuartzScheduler#standby - Scheduler TimediffScheduler_$_iZu1skaofy1Z1433914696931 paused.
26231 10-Jun-2015 14:15:35.072 INFO [localhost-startStop-2] org.springframework.scheduling.quartz.SchedulerFactoryBean.destroy Shutting down Quartz Scheduler
26232 2015-06-10 14:15:35.072 {localhost-startStop-2} INFO  org.quartz.core.QuartzScheduler#shutdown - Scheduler TimediffScheduler_$_iZu1skaofy1Z1433914696931 shutting down.
26233 2015-06-10 14:15:35.075 {localhost-startStop-2} INFO  org.quartz.core.QuartzScheduler#standby - Scheduler TimediffScheduler_$_iZu1skaofy1Z1433914696931 paused.
26234 2015-06-10 14:15:35.077 {localhost-startStop-2} INFO  org.quartz.core.QuartzScheduler#shutdown - Scheduler TimediffScheduler_$_iZu1skaofy1Z1433914696931 shutdown complete.
26235 10-Jun-2015 14:15:35.082 INFO [localhost-startStop-2] org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.shutdown Shutting down ExecutorService 'quartzThreadPool'
26236 2015-06-10 14:15:35.103 {localhost-startStop-2} INFO  com.timediff.listener.StopMemoryLeakListener#lambda$contextDestroyed$0 - driver: com.mysql.jdbc.Driver@7657b26d is de-registered.
26237 2015-06-10 14:15:35.104 {localhost-startStop-2} INFO  com.timediff.listener.StopMemoryLeakListener#contextDestroyed - AbandonedConnectionCleanupThread shutdown.
26238 10-Jun-2015 14:15:35.150 INFO [Thread-11] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8080"]
26239 10-Jun-2015 14:15:35.152 INFO [Thread-11] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-nio-8009"]
26240 10-Jun-2015 14:15:35.154 INFO [Thread-11] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8080"]
26241 10-Jun-2015 14:15:35.156 INFO [Thread-11] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["ajp-nio-8009"]

在 stackoverflow 上，this question和 this question和我的情况很相似，但我还是迷迷糊糊了。

现在我将详细描述我的问题:

2。更多详情

2.1 tomcat和jdk版本

Tomcat: 8.0.22
JDK: 1.8.0_45

catalina.sh 中的 2.2 jvm 选项:

CATALINA_OPTS="-server -Xms1g -Xmx1g -XX:MaxMetaspaceSize=512m -Xmn512m 
-XX:SurvivorRatio=8 
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70 -XX:+ScavengeBeforeFullGC 
-XX:+CMSScavengeBeforeRemark
-XX:+PrintGCDateStamps -verbose:gc -XX:+PrintGCDetails 
-Xloggc:/opt/logs/gc/timediff-gc.log 
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M
-Dsun.net.inetaddr.ttl=120  -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/opt/logs/gc/timediff-oom.hprof  
-Djava.rmi.server.hostname=**.**.**.**
-Dcom.sun.management.jmxremote.port=1099 
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false"

2.3 在我的 webapp 中没有与 tomcat 中止相关的异常日志，我确定我从未调用过 System.exit()，也没有像这样的代码块:

try {

} catch(Exception e) {
    // do nothing
}

2.4 而我实际上在 gc 日志中发现了 Allocation Failure:

2015-06-10T15:36:28.589+0800: 3099.795: [GC (Allocation Failure) 3099.795: [ParNew: 419780K->382K(471872K), 0.0125816 secs] 469721K->50348K(996160K), 0.0126820 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2015-06-10T15:37:30.141+0800: 3161.347: [GC (Allocation Failure) 3161.347: [ParNew: 419838K->372K(471872K), 0.0062445 secs] 469804K->50338K(996160K), 0.0063629 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2015-06-10T15:38:41.680+0800: 3232.886: [GC (Allocation Failure) 3232.886: [ParNew: 419828K->369K(471872K), 0.0064920 secs] 469794K->50356K(996160K), 0.0066009 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2015-06-10T15:39:43.222+0800: 3294.428: [GC (Allocation Failure) 3294.428: [ParNew: 419825K->384K(471872K), 0.0058772 secs] 469812K->50372K(996160K), 0.0059823 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2015-06-10T15:40:54.758+0800: 3365.964: [GC (Allocation Failure) 3365.964: [ParNew: 419840K->388K(471872K), 0.0056674 secs] 469828K->50395K(996160K), 0.0069850 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]

我认为可能是这个原因，但 TOP 和 jvisualVM 的结果使它不清楚:

web@iZu1skaofy1Z:/usr/local/apache-tomcat-8.0.22/logs$ free -m
             total       used       free     shared    buffers     cached
Mem:          3951       3087        864          0        190        553
-/+ buffers/cache:       2343       1608
Swap:            0          0          0

top - 15:50:05 up 16 days,  5:11,  2 users,  load average: 0.33, 0.17, 0.09
Tasks: 128 total,   2 running, 126 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.8 us,  0.5 sy,  0.0 ni, 98.5 id,  0.0 wa,  0.2 hi,  0.0 si,  0.0 st
KiB Mem:   4046820 total,  3161260 used,   885560 free,   194880 buffers
KiB Swap:        0 total,        0 used,        0 free.   566984 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
27307 web       20   0 2068604 865872  22048 S   0.7 21.4  20:20.28 java
16557 web       20   0 3680756 801708  13740 S   0.0 19.8   2:02.99 java
15597 mysql     20   0 1800972 526220   6636 S   0.0 13.0  36:26.08 mysqld

2.4 我在同一台服务器上部署了另一个tomcat，但是我更改了关闭端口和连接器端口，我认为它们不冲突。

我已经尽力了，可能是我在分析的过程中忘记了什么，请帮忙给我一些提示，在此先感谢!

update(2015-07-04):我运行tomcat时从web用户切换到root用户后，问题没有出现.所以我怀疑tomcat是不是因为用户权限被系统kill了，有知道的请告诉我，谢谢!

Answer 1

This answer (从你找到的一个问题)看起来不错。

有东西告诉 Tomcat 停止。而且因为当 Tomcat 以 root 运行时它不会发生，我认为原因是其他一些(非系统)进程(可能是脚本或 cron 作业)向 Tomcat 发送信号(可能是 SIGTERM )，例如 kill <tomcat pid> .也许其他进程也以用户身份运行 web - 这可以解释为什么该进程无法终止 root的 Tomcat 。或者其他进程可能只是搜索要终止的进程，其中一个条件是“web 拥有的进程”。

我建议你仔细阅读用户的crontabs root和 web 、系统范围的 crontab 和 /etc/cron.*/ 中的所有内容文件夹。您还可以检查 web 是否拥有任何其他进程突然终止。从源代码构建 Tomcat，添加一些跟踪(如我提到的答案中所建议的)，似乎是个好主意。