环境

• 生产环境应用服务假死,基于SpringCloud Dalston.SR4 + SpringBoot1.5.6.RELEASE + Tomcat8.5.16
• Linux环境

>$ uname -a
Linux tst-center-login-temp 3.10.0-1127.13.1.el7.x86_64 #1 SMP Tue Jun 23 15:46:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
复制

现象

• 应用服务出现假死，netstat发现该服务进程大量close_wait

$ netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
SYN_RECV 8
CLOSE_WAIT 7085
ESTABLISHED 272
FIN_WAIT1 1
FIN_WAIT2 3
TIME_WAIT 251
复制

• ss -lnt,tomcat设置的backlog为800，全连接被打满

>$ ss -lnt
State      Recv-Q Send-Q                                                                                       Local Address:Port                                                                                         Peer Address:Port
LISTEN     801    800                                                                                                      *:8210                                                                                                    *:*

复制

• jstack发现日志缺少ClientPoller，同时Acceptor到达连接上限阻塞在NioEndpointd的Acceptor.countUpOrAwaitConnection()

"http-nio-8210-Acceptor-0" #53 daemon prio=5 os_prio=0 tid=0x00007f36a8837000 nid=0xdc1 waiting on condition [0x00007f36636f5000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000856cfa08> (a org.apache.tomcat.util.threads.LimitLatch$Sync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
	at org.apache.tomcat.util.threads.LimitLatch.countUpOrAwait(LimitLatch.java:117)
	at org.apache.tomcat.util.net.AbstractEndpoint.countUpOrAwaitConnection(AbstractEndpoint.java:1074)
	at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:447)
	at java.lang.Thread.run(Thread.java:745)
复制

求助

从jstack日志中发现ClientPoller线程丢失，实在是百思不得其解，求求路过的好心人帮助下迷路的孩子