java - 由于线程处于 WAITING 和 TIMED_WAITING 状态，请求处于挂起状态

Question

Java Spring Boot 应用程序请求处于挂起状态，因为线程处于 WAITING 和 TIMED_WAITING 状态。

Jstack 日志:

"qtp886341817-1399" #1399 prio=5 os_prio=0 tid=0x00007f02142ae800 nid=0x22f904 waiting on condition [0x00007f01c3fa8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x0000000684588e00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
    at java.lang.Thread.run(Thread.java:748)

"threadPoolTaskExecutor-1" #114 prio=5 os_prio=0 tid=0x00007f02140b4800 nid=0x229d78 waiting on condition [0x00007f01c55b2000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x0000000684588e58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

"qtp886341817-717" #717 prio=5 os_prio=0 tid=0x00007f021c102000 nid=0x22c546 in Object.wait() [0x00007f01ee774000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at org.eclipse.paho.client.mqttv3.internal.Token.waitUntilSent(Token.java:248)
    - locked <0x0000000689516c80> 
(a java.lang.Object) at org.eclipse.paho.client.mqttv3.MqttTopic.publish(MqttTopic.java:117)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Thread.java:748)

详细信息:

1. 在这种情况下，应用程序无法提供API服务，当前线程数达到250+，许多线程处于死锁状态。

2. 此 Spring 应用程序托管在 AWS 的 t2.medium 实例上，Xms=1g、Xmx=2g、UseG1GC，我们正在使用 jetty 服务器。

3. 该应用程序通常提供长线程等待 API，响应某些 API 至少需要 12 到 60 秒。

问题:

1. 有什么方法可以找出 Spring 应用程序/JVM/jetty 服务器可以处理多少线程。

2. 我们如何调整此应用程序以避免出现这种情况(当应用程序无响应时)

3. 如何在出现这种挂起情况之前限制 API。

Answer 1

看这里:

at org.eclipse.paho.client.mqttv3.internal.Token.waitUntilSent(Token.java:248)
- locked <0x0000000689516c80> 

有一个锁定，发生在尝试发送消息超时时。尝试在此处添加异步调用。