java - 运行 Storm 拓扑时为 "java.lang.OutOfMemoryError: unable to create new native thread"

Question

我正在使用 kafkaspout 将消息从 kafka 队列读取到 Storm 拓扑中，它在过去一周运行良好，但从昨天开始，它给了我以下错误:内存不足。我已经检查了代码中所有与内存相关的参数和内存泄漏，但似乎一切都很好，

    java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method) ~[na:1.7.0_45]
    at java.lang.Thread.start(Thread.java:713) ~[na:1.7.0_45]
    at backtype.storm.util$async_loop.doInvoke(util.clj:421) ~[storm-core-0.9.0-rc3.jar:na]
    at clojure.lang.RestFn.invoke(RestFn.java:559) [clojure-1.4.0.jar:na]
    at backtype.storm.daemon.executor$eval9473$fn__9474.invoke(executor.clj:732) ~[na:na]
    at clojure.lang.MultiFn.invoke(MultiFn.java:167) ~[clojure-1.4.0.jar:na]
    at backtype.storm.daemon.executor$mk_executor$fn__9342.invoke(executor.clj:317) ~[na:na]
    at backtype.storm.daemon.executor$mk_executor.invoke(executor.clj:316) [na:na]
    at         backtype.storm.daemon.worker$eval9968$exec_fn__6466__auto____9969$iter__9974__9978$fn__9979.        invoke(worker.clj:360) [na:na]
    at clojure.lang.LazySeq.sval(LazySeq.java:42) [clojure-1.4.0.jar:na]
    at clojure.lang.LazySeq.seq(LazySeq.java:60) [clojure-1.4.0.jar:na]
    at clojure.lang.Cons.next(Cons.java:39) [clojure-1.4.0.jar:na]
    at clojure.lang.LazySeq.next(LazySeq.java:92) [clojure-1.4.0.jar:na]
    at clojure.lang.RT.next(RT.java:587) [clojure-1.4.0.jar:na]
    at clojure.core$next.invoke(core.clj:64) [na:na]
    at clojure.core$dorun.invoke(core.clj:2726) [na:na]
    at clojure.core$doall.invoke(core.clj:2741) [na:na]
    at   backtype.storm.daemon.worker$eval9968$exec_fn__6466__auto____9969.invoke(worker.clj:360)   [na:na]
    at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.4.0.jar:na]
    at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
    at clojure.core$apply.invoke(core.clj:601) [na:na]
    at backtype.storm.daemon.worker$eval9968$mk_worker__10024.doInvoke(worker.clj:329) [na:na]
    at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.4.0.jar:na]
    at backtype.storm.daemon.worker$_main.invoke(worker.clj:439) [na:na]
    at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.4.0.jar:na]
    at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
    at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.0-rc3.jar:na]
    2014-04-11 13:56:11 b.s.util [ERROR] Async loop died!
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method) [na:1.7.0_45]
    at java.lang.Thread.start(Thread.java:713) [na:1.7.0_45]
    at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) ~[na:1.7.0_45]
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360) ~[na:1.7.0_45]
    at      java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132) ~[na:1.7.0_45]
    at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:641) ~[na:1.7.0_45]
    at com.netflix.curator.framework.state.ConnectionStateManager.start(ConnectionStateManager.java:83) ~[curator-framework-1.0.1.jar:na]
    at com.netflix.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:168) ~[curator-framework-1.0.1.jar:na]
    at storm.kafka.DynamicBrokersReader.<init>(DynamicBrokersReader.java:35) ~[stormjar.jar:na]
    at storm.kafka.trident.ZkBrokerReader.<init>(ZkBrokerReader.java:23) ~[stormjar.jar:na]
    at storm.kafka.trident.KafkaUtils.makeBrokerReader(KafkaUtils.java:26) ~[stormjar.jar:na]
    at storm.kafka.KafkaSpout.open(KafkaSpout.java:67) ~[stormjar.jar:na]
    at backtype.storm.daemon.executor$eval9403$fn__9404$fn__9419.invoke(executor.clj:504) ~[na:na]
    at backtype.storm.util$async_loop$fn__5748.invoke(util.clj:401) ~[na:na]
    at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
    at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
    2014-04-11 13:56:11 b.s.d.executor [ERROR]
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method) [na:1.7.0_45]
    at java.lang.Thread.start(Thread.java:713) [na:1.7.0_45]
    at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) ~[na:1.7.0_45]
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360) ~[na:1.7.0_45]
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132) ~[na:1.7.0_45]
    at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:641) ~[na:1.7.0_45]
    at com.netflix.curator.framework.state.ConnectionStateManager.start(ConnectionStateManager.java:83) ~[curator-framework-1.0.1.jar:na]
    at com.netflix.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:168) ~[curator-framework-1.0.1.jar:na]
    at storm.kafka.DynamicBrokersReader.<init>(DynamicBrokersReader.java:35) ~[stormjar.jar:na]
    at storm.kafka.trident.ZkBrokerReader.<init>(ZkBrokerReader.java:23) ~[stormjar.jar:na]
    at storm.kafka.trident.KafkaUtils.makeBrokerReader(KafkaUtils.java:26) ~[stormjar.jar:na]
    at storm.kafka.KafkaSpout.open(KafkaSpout.java:67) ~[stormjar.jar:na]
    at backtype.storm.daemon.executor$eval9403$fn__9404$fn__9419.invoke(executor.clj:504) ~[na:na]
    at backtype.storm.util$async_loop$fn__5748.invoke(util.clj:401) ~[na:na]
    at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
    at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
    2014-04-11 13:56:11 b.s.util [INFO] Halting process: ("Worker died")
    2014-04-11 13:56:11 b.s.util [INFO] Halting process: ("Worker died")

我的storm集群参数是: nimbus.childopts-Xmx1024m worker .childopts-Xmx768m drpc.childopts -Xmx768m

Answer 1

无法创建新的 native 线程与堆无关，它实际上是来自操作系统的代理消息，操作系统无法分配更多 native 内存来在操作系统级别创建新线程和/或进程。