3回复
3年前
Java C2 CompilerThread 长期占用 CPU 过高,如何查找原因?
Docker 宿主机情况
- CPU,cat /proc/cpuinfo | grep processor | wc -l 为 40
- 内存(free -g):
- total:188
- used:187
- free: 1
- Docker version 17.12.0-ce, build c97c6d6
- 运行 Docker 容器个数 130 左右
- Docker 容器运行时没有 CPU 和 Mem 资源限制
- Docker 容器内 JDK 最大堆统一为 2G
Docker 容器内部情况
- JDK: 1.8.0_281,HotSpot™ 64-Bit Server VM (build 25.281-b09, mixed mode)
- Web 容器 Resin 4.0.62
- 部分 JVM 参数:
-d64 -Xss1024k -Xmx2048M -Xms2048M -XX:MaxMetaspaceSize=256M -XX:MetaspaceSize=256M -XX:NewRatio=2 -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses -XX:+CMSClassUnloadingEnabled -XX:+ParallelRefProcEnabled -XX:+CMSScavengeBeforeRemark -XX:ParallelGCThreads=4 -XX:SurvivorRatio=3 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15 -XX:InitialCodeCacheSize=50M -XX:ReservedCodeCacheSize=64M -XX:+UseCodeCacheFlushing -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC
- 访问情况: 线下环境,几乎没有什么访问量
表现情况
- C2 CompilerThread 长期占用 CPU 高(95% 以上)
- 并非占用所有 CPU 资源,只会占用一两个 CPU 资源
- 不一定是某个应用,出现问题的应用不确定
- 并非一定出现,重启后会恢复正常,无法复现
- 整体服务的整体性能无影响,不影响响应速度
相关截图
Docker 容器内 top(>100% 进程占用的 CPU 的 百分比,)
top -Hp 184
使用 Arthas 查看具体是哪个线程(C2)
CodeCache
hsdb 线程信息
hsdb> thread 225
Thread 225 Address 0x00007ffaf5998800
State: IN_NATIVE
No Java frames present
Base of Stack: 0x00007ffaa5be9000
Last_Java_SP: null
Last_Java_FP: null
Last_Java_PC: null
Thread id: 225
hsdb> inspect 0x00007ffaf5998800
Type is CompilerThread (size of 1032)
oop ThreadShadow::_pending_exception: null null
char* ThreadShadow::_exception_file: char @ null
int ThreadShadow::_exception_line: 0
uint32_t Thread::_suspend_flags: 0
JNIHandleBlock* Thread::_active_handles: JNIHandleBlock @ 0x00007ffa640464d0
ThreadLocalAllocBuffer Thread::_tlab: ThreadLocalAllocBuffer @ 0x00007ffaf5998858
jlong Thread::_allocated_bytes: 304
ObjectMonitor* Thread::_current_pending_monitor: ObjectMonitor @ null
bool Thread::_current_pending_monitor_is_from_java: 1
ObjectMonitor* Thread::_current_waiting_monitor: ObjectMonitor @ null
ResourceArea* Thread::_resource_area: ResourceArea @ 0x00007ffaf5995540
JavaThread* JavaThread::_next: JavaThread @ 0x00007ffa6c001000
oop JavaThread::_hreadObj: Oop for java/lang/Thread @ 0x00000000aabbb9b0 Oop for java/lang/Thread @ 0x00000000aabbb9b0
JavaFrameAnchor JavaThread::_anchor: JavaFrameAnchor @ 0x00007ffaf59989d8
oop JavaThread::_vm_result: null null
Metadata* JavaThread::_vm_result_2: Metadata @ null
oop JavaThread::_pending_async_exception: null null
oop JavaThread::_exception_oop: null null
address JavaThread::_exception_pc: address @ 0x00007ffaf5998ab0
int JavaThread::_is_method_handle_return: 0
JavaThread::AsyncRequests JavaThread::_special_runtime_exit_condition: 0
address JavaThread::_saved_exception_pc: address @ 0x00007ffaf5998a80
JavaThreadState JavaThread::_thread_state: 4
OSThread* JavaThread::_osthread: OSThread @ 0x00007ffaf5995730
address JavaThread::_stack_base: address @ 0x00007ffaf5998948
size_t JavaThread::_stack_size: 1052672
vframeArray* JavaThread::_vframe_array_head: vframeArray @ null
vframeArray* JavaThread::_vframe_array_last: vframeArray @ null
ObjPtrQueue JavaThread::_satb_mark_queue: ObjPtrQueue @ 0x00007ffaf5998ae0
DirtyCardQueue JavaThread::_dirty_card_queue: DirtyCardQueue @ 0x00007ffaf5998b20
ciEnv* CompilerThread::_env: ciEnv @ 0x00007ffaa5be7a40
问题
- 访问量不大,应该没有热点代码,C2 编译器为什么会长期占用 CPU
- 如何排查 C2 长期占用 CPU 的原因,以及如何复现
- 是否是 JDK 的 Bug,JDK 从 1.8.0_151281 换到 1.8.0_281,还是有部分服务会出现该问题
一个类似的问题:
https://medium.com/@vladimirsitniko/analyzing-a-stuck-hotspot-c2-compilation-85e0ca230744
7921 阅读