JVM CPU 彪高问题排查

小半 • 2023年1月15日下午9:20 • 技术随笔 • 阅读 288

环境

Spring Cloud 、 K8s 、 Rancher 管理k8s 、服务 Continue 1核2G。

排查记录

因为是k8s的环境没法安装Arthas ，也没暴露端口出来，Arthas也没法远程链接。
同时也没法（没权限）执行 top 、top -Hp pid 命令查看线程占用。

最终解决方案。到Spring Boot Admin中查看项目的JVM信息。

多次下载线程信息对比发现。一个 scheduling-1 线程一直在执行并处在同一个代码处。去看日志确定确实是当前方法的13万条数据一直在执行，一直没让出CPU，一个方法执行了俩天了，数据一直在一个for循环里 continue ，导致没有IO操作让CPU歇一歇，CPU一直处于疯狂工作。

"scheduling-1" - Thread t@65
   java.lang.Thread.State: RUNNABLE
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:514)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:501)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:487)
	at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:241)
	at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:531)
	at 业务代码(业务代码.java:240)
	at 业务代码(业务代码.java:50)
	at sun.reflect.GeneratedMethodAccessor517.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
	at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:819)

   Locked ownable synchronizers:
	- Locked <4bd7db25> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"DestroyJavaVM helper thread" - Thread t@67
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None