spring-boot - 修复 "The web application [ROOT] created a ThreadLocal with key of type [com.netflix.hystrix.Hystrix$1]"

Question

我在 spring-boot API 网关中使用 Netflix 云，并将 spring-boot 应用程序构建为 WAR，可以独立运行或部署到 Tomcat 容器。例如，当使用 Cargo Maven 插件在 Tomcat 中重新部署 spring-boot 应用程序时，Tomcat MemoryLeakDetection 会提示“Web 应用程序 [ROOT] 创建了一个 key 类型为 [com.netflix.hystrix.Hystrix$1] 的 ThreadLocal”。

重新部署我的 spring-boot 应用程序 9 次后，Tomcat 内存不足。如何删除 Hystrix ThreadLocal，以便它不会导致每次重新部署 spring-boot 应用程序时 WebappClassLoader 实例都保留下来。结果是每次重新部署都会留下一个 WebappClassLoader 实例，由于 Hystrix ThreadLocal 而无法对其进行垃圾收集？

这是内存不足的堆栈跟踪:

29-Apr-2016 12:21:50.721 SEVERE [http-apr-8080-exec-38] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [my-spring-boot-app] created a ThreadLocal with key of type [com.netflix.hystrix.Hystrix$1] (value [com.netflix.hystrix.Hystrix$1@8302924]) and a value of type [java.util.LinkedL
ist] (value [[]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
29-Apr-2016 12:22:00.484 INFO [http-apr-8080-exec-38] org.apache.catalina.startup.HostConfig.undeploy Undeploying context [/my-spring-boot-app]
29-Apr-2016 12:22:09.353 INFO [http-apr-8080-exec-33] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive D:\apache-tomcat-8.0.28\webapps\my-spring-boot-app.war
12:23:08,056 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
12:23:08,089 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Setting ReconfigureOnChangeFilter scanning period to 30 seconds
12:23:08,089 |-INFO in ReconfigureOnChangeFilter{invocationCounter=0} - Will scan for changes in [[D:\apache-tomcat-8.0.28\webapps\my-spring-boot-app\WEB-INF\classes\logba
ck.xml]] every 30 seconds.
12:23:08,089 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Adding ReconfigureOnChangeFilter as a turbo filter
12:23:08,106 |-INFO in ch.qos.logback.core.joran.action.StatusListenerAction - Added status listener of type [ch.qos.logback.core.status.OnConsoleStatusListener]
12:23:08,118 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
12:23:08,136 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [CONSOLE]
12:23:08,176 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
java.lang.OutOfMemoryError: Metaspace
Dumping heap to java_pid15972.hprof ...
Heap dump file created [206407579 bytes in 1.201 secs]
29-Apr-2016 12:23:10.870 SEVERE [http-apr-8080-exec-33] org.apache.tomcat.util.modeler.BaseModelMBean.invoke Exception invoking method check
java.lang.OutOfMemoryError: Metaspace
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2496)
        at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:860)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1302)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1167)
        at org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:97)
        at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
        at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
        at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
        at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
        at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
        at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
        at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:655)
        at org.springframework.boot.context.web.SpringBootServletInitializer.onStartup(SpringBootServletInitializer.java:84)
        at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:175)

Answer 1

好吧，我想我解决了我自己的问题，这样 Tomcat MemoryLeakDetection 就不再提示 Hystrix ThreadLocal 了。我所做的是添加了一个自定义 ServletContextListener，并在“contextDestroyed()”方法中找到了一种方法来强制从 com.netflix.hystrix.Hystrix 类中删除私有(private)静态 ThreadLocal。这似乎可以解决问题。以下是我的自定义 servlet 监听器的摘录:

/**
 * The listener interface for receiving ServletContext events.
 * The class that is interested in processing a ServletContext
 * event implements this interface, and the object created
 * with that class is registered with a component using the
 * component's <code>addServletContextListener<code> method. When
 * the ServletContext event occurs, that object's appropriate
 * method is invoked.
 *
 * @see ServletContextEvent
 */
@Component
public class GatewayServletContextListener implements ServletContextListener{

    private static final Logger LOG = LoggerFactory.getLogger(GatewayServletContextListener.class);

    @Override
    public void contextInitialized(ServletContextEvent arg0) {
        LOG.info("Servlet context listener observed context initialized");
    }

    @Override
    public void contextDestroyed(ServletContextEvent arg0) {
        LOG.info("Servlet context listener observed context destroyed");
        cleanupThreadLocals();
    }



    /**
     * Cleanup thread locals.
     */
    private void cleanupThreadLocals() {

        try {
                LOG.info("Cleaning up ThreadLocals ...");

                Field currentCommandField = ReflectionUtils.findField(Hystrix.class, "currentCommand");
                Preconditions.checkNotNull(currentCommandField);

                ReflectionUtils.makeAccessible(currentCommandField);

                @SuppressWarnings("rawtypes")
                ThreadLocal currentCommand = (ThreadLocal)currentCommandField.get(null);

                Preconditions.checkNotNull(currentCommand);

                currentCommand.remove();

                LOG.info("Forcibly removed Hystrix 'currentCommand' ThreadLocal");
                LOG.info("Done cleaning up ThreadLocals");

            } catch(Exception e) {
                LOG.warn(e.getMessage(), e);
            }

    }

}

这是我的 spring-boot 应用程序重新部署的 Tomcat 日志:

2016-05-03/10:11:08.646/PDT [http-apr-8080-exec-3] INFO  x.y.z.listeners.GatewayServletContextListener - Servlet context listener observed context destroyed
2016-05-03/10:11:08.648/PDT [http-apr-8080-exec-3] INFO  x.y.z.listeners.GatewayServletContextListener - Cleaning up ThreadLocals ...
2016-05-03/10:11:08.652/PDT [http-apr-8080-exec-3] INFO  x.y.z.listeners.GatewayServletContextListener - Forcibly removed Hystrix 'currentCommand' ThreadLocal
2016-05-03/10:11:08.654/PDT [http-apr-8080-exec-3] INFO  x.y.z.listeners.GatewayServletContextListener - Done cleaning up ThreadLocals