4回复
5年前
jvm进程没了
线上使用的jdk为
java version “1.8.0_25”
Java™ SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot™ 64-Bit Server VM (build 25.25-b02, mixed mode)
然后使用的jetty容器。我们新上了一个agent，然后接入的应用过段时间就会挂掉，没有kill日志，时间也没有规律，后面看进程RSS占用很高，测试去掉agent依然很高，一直怀疑是内存被堆外吃掉，然后有抖动造成被kill，但是为啥没有kill日志呢？（这里有虚拟机也有容器），后面找了一些资料看是jdk1.8的bug，把jetty对servlet3.0 的注解支持给去掉了，RSS就下来了，原来再2.3-3.0G之间现在能稳定在1G左右。但是进程一段时间依然会挂。。原来一直忽略了jvmerror文件,这次看了下，
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000000000000000, pid=11735, tid=140251454990080
#
# JRE version: Java(TM) SE Runtime Environment (8.0_25-b17) (build 1.8.0_25-b17)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.25-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C 0x0000000000000000
#
# Core dump written. Default location: /home/hrs-common-data-gops/deploy/jettybase/core or core.11735
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x00000000044fc800): JavaThread "qtp1256728724-21" [_thread_in_Java, id=11985, stack(0x00007f8ed6236000,0x00007f8
ed6277000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000
Registers:
RAX=0x00000000d93a90f8, RBX=0x00007f8ede5c2a68, RCX=0x00007f8ee21d8d00, RDX=0x00000000d858a998
RSP=0x00007f8ed6271ac0, RBP=0x3d2ef35793c76730, RSI=0x00000000d93a90f8, RDI=0x000000010018c2e0
R8 =0x00000000d858a998, R9 =0x00000000d94cc128, R10=0x00000000d93a90f8, R11=0x00007f8ee3c269a0
R12=0x0000000000000000, R13=0x00007f8ed6271ab0, R14=0x00007f8ed6271b58, R15=0x00000000044fc800
RIP=0x0000000000000000, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000014
TRAPNO=0x000000000000000e
Top of Stack: (sp=0x00007f8ed6271ac0)
0x00007f8ed6271ac0: 0000000000000000 0000000000000000
0x00007f8ed6271ad0: 00007f8ed6271b40 00007f8ee216b710
0x00007f8ed6271ae0: 3fe62e42fefa3800 00007f8ee216b710
0x00007f8ed6271af0: 00000000d93a90f8 00000000d858da18
0x00007f8ed6271b00: 00007f8ed6271b00 00007f8ece7461e6
0x00007f8ed6271b10: 00007f8ed6271b58 00007f8ece746268
0x00007f8ed6271b20: 0000000000000000 00007f8ece7461f8
0x00007f8ed6271b30: 00007f8ed6271af0 00007f8ed6271b60
0x00007f8ed6271b40: 00007f8ed6271b60 00007f8ee2f4c584
0x00007f8ed6271b50: 00000000d93a90f8 00000000d92f35a8
0x00007f8ed6271b60: 00000000d844bd28 00007f8ed6271b68
0x00007f8ed6271b70: 00007f8ed49b6128 00007f8ed6271bb8
0x00007f8ed6271b80: 00007f8ed3746070 0000000000000000
0x00007f8ed6271b90: 00007f8ed6271c20 00007f8ee216b8b8
0x00007f8ed6271ba0: 00007f8ed6271bc0 00007f8ee216b8b8
0x00007f8ed6271bb0: 00000000d92f35a8 00000000d844bd28
0x00007f8ed6271bc0: 00000000d8e93bf0 0000000000000001
0x00007f8ed6271bd0: 00000000cb8666c8 00000000cb8666c8
0x00007f8ed6271be0: 00007f8ed6271be0 00007f8ed217f23b
0x00007f8ed6271bf0: 00007f8ed6271c30 00007f8ed217f568
0x00007f8ed6271c00: 0000000000000000 00007f8ed217f250
0x00007f8ed6271c10: 00007f8ed6271bb0 00007f8ed6271c30
0x00007f8ed6271c20: 00007f8ed6271c80 00007f8ee216b710
0x00007f8ed6271c30: 0000000000000001 00000000cb866688
0x00007f8ed6271c40: 00007f8ed6271c40 00007f8ed536595a
0x00007f8ed6271c50: 00007f8ed6271c90 00007f8ed536a390
0x00007f8ed6271c60: 0000000000000000 00007f8ed5365970
0x00007f8ed6271c70: 00007f8ed6271c30 00007f8ed6271c90
0x00007f8ed6271c80: 00007f8ed6271cd8 00007f8ee216b7e4
0x00007f8ed6271c90: 00000000cb866688 00007f8ed6271c98
0x00007f8ed6271ca0: 00007f8ed21186a9 00007f8ed6271cf8
0x00007f8ed6271cb0: 00007f8ed124aa38 0000000000000000
Instructions: (pc=0x0000000000000000)
0xffffffffffffffe0:
Register to memory mapping:
RAX=0x00000000d93a90f8 is an oop
java.util.stream.FindOps$$Lambda$47/2083004088
- klass: 'java/util/stream/FindOps$$Lambda$47'
RBX={method} {0x00007f8ede5c2a68} 'identity' '(Ljava/lang/Object;)Ljava/lang/Object;' in 'sun/invoke/util/ValueConversions'
RCX=0x00007f8ee21d8d00 is at begin+12 in a stub
MethodHandle::interpreter_entry::_linkToStatic [0x00007f8ee21d8cf4, 0x00007f8ee21d8d18[ (36 bytes)
RDX=0x00000000d858a998 is an oop
java.lang.invoke.MemberName
- klass: 'java/lang/invoke/MemberName'
RSP=0x00007f8ed6271ac0 is pointing into the stack for thread: 0x00000000044fc800
RBP=0x3d2ef35793c76730 is an unknown value
RSI=0x00000000d93a90f8 is an oop
java.util.stream.FindOps$$Lambda$47/2083004088
- klass: 'java/util/stream/FindOps$$Lambda$47'
RDI=0x000000010018c2e0 is pointing into metadata
R8 =0x00000000d858a998 is an oop
java.lang.invoke.MemberName
- klass: 'java/lang/invoke/MemberName'
R9 =0x00000000d94cc128 is an oop
[C
- klass: {type array char}
- length: 7
R10=0x00000000d93a90f8 is an oop
java.util.stream.FindOps$$Lambda$47/2083004088
- klass: 'java/util/stream/FindOps$$Lambda$47'
R11=0x00007f8ee3c269a0 is at entry_point+0 in (nmethod*)0x00007f8ee3c26850
R12=0x0000000000000000 is an unknown value
R13=0x00007f8ed6271ab0 is pointing into the stack for thread: 0x00000000044fc800
R14=0x00007f8ed6271b58 is pointing into the stack for thread: 0x00000000044fc800
R15=0x00000000044fc800 is a thread
Stack: [0x00007f8ed6236000,0x00007f8ed6277000], sp=0x00007f8ed6271ac0, free space=238k
--------------- P R O C E S S ---------------
信息太多贴部分
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap:
par new generation total 353920K, used 216729K [0x00000000c0000000, 0x00000000d8000000, 0x00000000d8000000)
eden space 314624K, 60% used [0x00000000c0000000, 0x00000000cba6e248, 0x00000000d3340000)
from space 39296K, 65% used [0x00000000d3340000, 0x00000000d4c783e0, 0x00000000d59a0000)
to space 39296K, 0% used [0x00000000d59a0000, 0x00000000d59a0000, 0x00000000d8000000)
concurrent mark-sweep generation total 655360K, used 29104K [0x00000000d8000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 82696K, capacity 84236K, committed 84480K, reserved 1122304K
class space used 10399K, capacity 10719K, committed 10752K, reserved 1048576K
Card table byte_map: [0x00007f8ee179f000,0x00007f8ee19a0000] byte_map_base: 0x00007f8ee119f000
Marking Bits: (CMSBitMap*) 0x00000000015b0348
Bits: [0x00007f8ee0bdd000, 0x00007f8ee15dd000)
Mod Union Table: (CMSBitMap*) 0x00000000015b0408
Bits: [0x00007f8ee0bb4000, 0x00007f8ee0bdc000)
Polling page: 0x00007f8ef32bc000
CodeCache: size=245760Kb used=31878Kb max_used=31879Kb free=213881Kb
bounds [0x00007f8ee2164000, 0x00007f8ee40b4000, 0x00007f8ef1164000]
total_blobs=8819 nmethods=8135 adapters=596
compilation: enabled
jvm 参数
VM Arguments:
jvm_args: -Xmx1024M -Xms1024M -Xmn384M -XX:MaxMetaspaceSize=256M -XX:MetaspaceSize=256M -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiat
ingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses -XX:+CMSClassUnloadingEna
bled -XX:+ParallelRefProcEnabled -XX:+CMSScavengeBeforeRemark -XX:-HeapDumpOnOutOfMemoryError -XX:+UseFastAccessorMethods -Xss256k
-XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:GCTimeRatio=19 -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:SoftRefLRUPo
licyMSPerMB=0 -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dapplication.codeset=UTF-8 -Dfile.encoding=UTF-8 -Ddubbo.a
pplication.qos.enable=false -XX:ErrorFile=/home/hrs-common-data-gops/jvmerror.log -XX:NativeMemoryTracking=detail -Xloggc:/home/hr
s-common-data-gops/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:He
apDumpPath=/home/hrs-common-data-gops/dump.hprof -javaagent:/home/hrs-common-data-gops/deploy/htdocs/war/gtrace-agent/gtrace-agent
.jar -Djetty.logging.dir=/data/logs/hrs-common-data-gops -Djetty.home=/usr/local/env/jetty9.3.9.v20160713 -Djetty.base=/home/hrs-c
ommon-data-gops/deploy/jettybase -Djava.io.tmpdir=/tmp
java_command: /usr/local/env/jetty9.3.9.v20160713/start.jar jetty.state=/home/hrs-common-data-gops/deploy/jettybase/jetty.state je
tty-logging.xml jetty-started.xml
java_class_path (initial): /usr/local/env/jetty9.3.9.v20160713/start.jar:/home/hrs-common-data-gops/deploy/htdocs/war/gtrace-agent
/gtrace-agent.jar
Launcher Type: SUN_STANDARD
内存情况
--------------- S Y S T E M ---------------
OS:CentOS release 6.10 (Final)
uname:Linux 2.6.32-754.22.1.el6.x86_64 #1 SMP Tue Sep 17 16:24:44 UTC 2019 x86_64
libc:glibc 2.12 NPTL 2.12
rlimit: STACK 10240k, CORE infinity, NPROC 15217, NOFILE 100002, AS infinity
load average:0.16 0.10 0.08
/proc/meminfo:
MemTotal: 4056072 kB
MemFree: 1997444 kB
Buffers: 286448 kB
Cached: 565444 kB
SwapCached: 0 kB
Active: 1384492 kB
Inactive: 307864 kB
Active(anon): 844572 kB
Inactive(anon): 376 kB
Active(file): 539920 kB
Inactive(file): 307488 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 52 kB
Writeback: 0 kB
AnonPages: 840476 kB
Mapped: 29388 kB
Shmem: 4484 kB
Slab: 312688 kB
SReclaimable: 287820 kB
SUnreclaim: 24868 kB
KernelStack: 4448 kB
PageTables: 6184 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 2028036 kB
Committed_AS: 1724476 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 18732 kB
VmallocChunk: 34359708124 kB
HardwareCorrupted: 0 kB
AnonHugePages: 692224 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 12144 kB
DirectMap2M: 2084864 kB
DirectMap1G: 2097152 kB
这里有看到上面的
- klass: 'java/util/stream/FindOps$$Lambda$47’信息
搜索了下看有小伙伴说是jdk1.8.0.25 lambada的一个bug升级最新1.8版本可解决，然后升级了下，是暂时还没有挂掉，但是想请教大牛怎么确认下这个问题，
最好有详细的思路和具体的工具使用方法（小白没啥经验，看到一些分析的文章总是被搁浅到怎么使用上，都是直接略过拿到了结果，可能对小白来说最重要的还是怎么使用，怎么定位。。）
这里也希望大牛能给一下经验，需要积累学习那些（路径？）才能对类似问题有思路（这里能很多东西原来压根就不知道，就很难有思路去排查，确认），万分感谢！
5090 阅读
