性能问答>jvm进程没了>
4回复
2年前

jvm进程没了



线上使用的jdk为
java version “1.8.0_25”
Java™ SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot™ 64-Bit Server VM (build 25.25-b02, mixed mode)
然后使用的jetty容器。我们新上了一个agent,然后接入的应用过段时间就会挂掉,没有kill日志,时间也没有规律,后面看进程RSS占用很高,测试去掉agent依然很高,一直怀疑是内存被堆外吃掉,然后有抖动造成被kill,但是为啥没有kill日志呢?(这里有虚拟机也有容器),后面找了一些资料看是jdk1.8的bug,把jetty对servlet3.0 的注解支持给去掉了,RSS就下来了,原来再2.3-3.0G之间现在能稳定在1G左右。但是进程一段时间依然会挂。。原来一直忽略了jvmerror文件,这次看了下,

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000000000000, pid=11735, tid=140251454990080
#
# JRE version: Java(TM) SE Runtime Environment (8.0_25-b17) (build 1.8.0_25-b17)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.25-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  0x0000000000000000
#
# Core dump written. Default location: /home/hrs-common-data-gops/deploy/jettybase/core or core.11735
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00000000044fc800):  JavaThread "qtp1256728724-21" [_thread_in_Java, id=11985, stack(0x00007f8ed6236000,0x00007f8
ed6277000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000

Registers:
RAX=0x00000000d93a90f8, RBX=0x00007f8ede5c2a68, RCX=0x00007f8ee21d8d00, RDX=0x00000000d858a998
RSP=0x00007f8ed6271ac0, RBP=0x3d2ef35793c76730, RSI=0x00000000d93a90f8, RDI=0x000000010018c2e0
R8 =0x00000000d858a998, R9 =0x00000000d94cc128, R10=0x00000000d93a90f8, R11=0x00007f8ee3c269a0
R12=0x0000000000000000, R13=0x00007f8ed6271ab0, R14=0x00007f8ed6271b58, R15=0x00000000044fc800
RIP=0x0000000000000000, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000014
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f8ed6271ac0)
0x00007f8ed6271ac0:   0000000000000000 0000000000000000
0x00007f8ed6271ad0:   00007f8ed6271b40 00007f8ee216b710
0x00007f8ed6271ae0:   3fe62e42fefa3800 00007f8ee216b710
0x00007f8ed6271af0:   00000000d93a90f8 00000000d858da18
0x00007f8ed6271b00:   00007f8ed6271b00 00007f8ece7461e6
0x00007f8ed6271b10:   00007f8ed6271b58 00007f8ece746268
0x00007f8ed6271b20:   0000000000000000 00007f8ece7461f8
0x00007f8ed6271b30:   00007f8ed6271af0 00007f8ed6271b60
0x00007f8ed6271b40:   00007f8ed6271b60 00007f8ee2f4c584
0x00007f8ed6271b50:   00000000d93a90f8 00000000d92f35a8
0x00007f8ed6271b60:   00000000d844bd28 00007f8ed6271b68
0x00007f8ed6271b70:   00007f8ed49b6128 00007f8ed6271bb8
0x00007f8ed6271b80:   00007f8ed3746070 0000000000000000
0x00007f8ed6271b90:   00007f8ed6271c20 00007f8ee216b8b8
0x00007f8ed6271ba0:   00007f8ed6271bc0 00007f8ee216b8b8
0x00007f8ed6271bb0:   00000000d92f35a8 00000000d844bd28
0x00007f8ed6271bc0:   00000000d8e93bf0 0000000000000001
0x00007f8ed6271bd0:   00000000cb8666c8 00000000cb8666c8
0x00007f8ed6271be0:   00007f8ed6271be0 00007f8ed217f23b
0x00007f8ed6271bf0:   00007f8ed6271c30 00007f8ed217f568
0x00007f8ed6271c00:   0000000000000000 00007f8ed217f250
0x00007f8ed6271c10:   00007f8ed6271bb0 00007f8ed6271c30
0x00007f8ed6271c20:   00007f8ed6271c80 00007f8ee216b710
0x00007f8ed6271c30:   0000000000000001 00000000cb866688
0x00007f8ed6271c40:   00007f8ed6271c40 00007f8ed536595a
0x00007f8ed6271c50:   00007f8ed6271c90 00007f8ed536a390
0x00007f8ed6271c60:   0000000000000000 00007f8ed5365970
0x00007f8ed6271c70:   00007f8ed6271c30 00007f8ed6271c90
0x00007f8ed6271c80:   00007f8ed6271cd8 00007f8ee216b7e4
0x00007f8ed6271c90:   00000000cb866688 00007f8ed6271c98
0x00007f8ed6271ca0:   00007f8ed21186a9 00007f8ed6271cf8
0x00007f8ed6271cb0:   00007f8ed124aa38 0000000000000000 

Instructions: (pc=0x0000000000000000)
0xffffffffffffffe0:   

Register to memory mapping:

RAX=0x00000000d93a90f8 is an oop
java.util.stream.FindOps$$Lambda$47/2083004088 
 - klass: 'java/util/stream/FindOps$$Lambda$47'
RBX={method} {0x00007f8ede5c2a68} 'identity' '(Ljava/lang/Object;)Ljava/lang/Object;' in 'sun/invoke/util/ValueConversions'
RCX=0x00007f8ee21d8d00 is at begin+12 in a stub
MethodHandle::interpreter_entry::_linkToStatic [0x00007f8ee21d8cf4, 0x00007f8ee21d8d18[ (36 bytes)
RDX=0x00000000d858a998 is an oop
java.lang.invoke.MemberName 
 - klass: 'java/lang/invoke/MemberName'
RSP=0x00007f8ed6271ac0 is pointing into the stack for thread: 0x00000000044fc800
RBP=0x3d2ef35793c76730 is an unknown value
RSI=0x00000000d93a90f8 is an oop
java.util.stream.FindOps$$Lambda$47/2083004088 
 - klass: 'java/util/stream/FindOps$$Lambda$47'
RDI=0x000000010018c2e0 is pointing into metadata
R8 =0x00000000d858a998 is an oop
java.lang.invoke.MemberName 
 - klass: 'java/lang/invoke/MemberName'
R9 =0x00000000d94cc128 is an oop
[C 
 - klass: {type array char}
 - length: 7
R10=0x00000000d93a90f8 is an oop
java.util.stream.FindOps$$Lambda$47/2083004088 
 - klass: 'java/util/stream/FindOps$$Lambda$47'
R11=0x00007f8ee3c269a0 is at entry_point+0 in (nmethod*)0x00007f8ee3c26850
R12=0x0000000000000000 is an unknown value
R13=0x00007f8ed6271ab0 is pointing into the stack for thread: 0x00000000044fc800
R14=0x00007f8ed6271b58 is pointing into the stack for thread: 0x00000000044fc800
R15=0x00000000044fc800 is a thread


Stack: [0x00007f8ed6236000,0x00007f8ed6277000],  sp=0x00007f8ed6271ac0,  free space=238k

---------------  P R O C E S S  ---------------

信息太多贴部分


VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap:
 par new generation   total 353920K, used 216729K [0x00000000c0000000, 0x00000000d8000000, 0x00000000d8000000)
  eden space 314624K,  60% used [0x00000000c0000000, 0x00000000cba6e248, 0x00000000d3340000)
  from space 39296K,  65% used [0x00000000d3340000, 0x00000000d4c783e0, 0x00000000d59a0000)
  to   space 39296K,   0% used [0x00000000d59a0000, 0x00000000d59a0000, 0x00000000d8000000)
 concurrent mark-sweep generation total 655360K, used 29104K [0x00000000d8000000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 82696K, capacity 84236K, committed 84480K, reserved 1122304K
  class space    used 10399K, capacity 10719K, committed 10752K, reserved 1048576K

Card table byte_map: [0x00007f8ee179f000,0x00007f8ee19a0000] byte_map_base: 0x00007f8ee119f000

Marking Bits: (CMSBitMap*) 0x00000000015b0348
 Bits: [0x00007f8ee0bdd000, 0x00007f8ee15dd000)

Mod Union Table: (CMSBitMap*) 0x00000000015b0408
 Bits: [0x00007f8ee0bb4000, 0x00007f8ee0bdc000)

Polling page: 0x00007f8ef32bc000

CodeCache: size=245760Kb used=31878Kb max_used=31879Kb free=213881Kb
 bounds [0x00007f8ee2164000, 0x00007f8ee40b4000, 0x00007f8ef1164000]
 total_blobs=8819 nmethods=8135 adapters=596
 compilation: enabled

jvm 参数


VM Arguments:
jvm_args: -Xmx1024M -Xms1024M -Xmn384M -XX:MaxMetaspaceSize=256M -XX:MetaspaceSize=256M -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiat
ingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses -XX:+CMSClassUnloadingEna
bled -XX:+ParallelRefProcEnabled -XX:+CMSScavengeBeforeRemark -XX:-HeapDumpOnOutOfMemoryError -XX:+UseFastAccessorMethods -Xss256k
 -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:GCTimeRatio=19 -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:SoftRefLRUPo
licyMSPerMB=0 -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dapplication.codeset=UTF-8 -Dfile.encoding=UTF-8 -Ddubbo.a
pplication.qos.enable=false -XX:ErrorFile=/home/hrs-common-data-gops/jvmerror.log -XX:NativeMemoryTracking=detail -Xloggc:/home/hr
s-common-data-gops/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:He
apDumpPath=/home/hrs-common-data-gops/dump.hprof -javaagent:/home/hrs-common-data-gops/deploy/htdocs/war/gtrace-agent/gtrace-agent
.jar -Djetty.logging.dir=/data/logs/hrs-common-data-gops -Djetty.home=/usr/local/env/jetty9.3.9.v20160713 -Djetty.base=/home/hrs-c
ommon-data-gops/deploy/jettybase -Djava.io.tmpdir=/tmp 
java_command: /usr/local/env/jetty9.3.9.v20160713/start.jar jetty.state=/home/hrs-common-data-gops/deploy/jettybase/jetty.state je
tty-logging.xml jetty-started.xml
java_class_path (initial): /usr/local/env/jetty9.3.9.v20160713/start.jar:/home/hrs-common-data-gops/deploy/htdocs/war/gtrace-agent
/gtrace-agent.jar
Launcher Type: SUN_STANDARD

内存情况

---------------  S Y S T E M  ---------------

OS:CentOS release 6.10 (Final)

uname:Linux 2.6.32-754.22.1.el6.x86_64 #1 SMP Tue Sep 17 16:24:44 UTC 2019 x86_64
libc:glibc 2.12 NPTL 2.12 
rlimit: STACK 10240k, CORE infinity, NPROC 15217, NOFILE 100002, AS infinity
load average:0.16 0.10 0.08

/proc/meminfo:
MemTotal:        4056072 kB
MemFree:         1997444 kB
Buffers:          286448 kB
Cached:           565444 kB
SwapCached:            0 kB
Active:          1384492 kB
Inactive:         307864 kB
Active(anon):     844572 kB
Inactive(anon):      376 kB
Active(file):     539920 kB
Inactive(file):   307488 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                52 kB
Writeback:             0 kB
AnonPages:        840476 kB
Mapped:            29388 kB
Shmem:              4484 kB
Slab:             312688 kB
SReclaimable:     287820 kB
SUnreclaim:        24868 kB
KernelStack:        4448 kB
PageTables:         6184 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     2028036 kB
Committed_AS:    1724476 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       18732 kB
VmallocChunk:   34359708124 kB
HardwareCorrupted:     0 kB
AnonHugePages:    692224 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       12144 kB
DirectMap2M:     2084864 kB
DirectMap1G:     2097152 kB

这里有看到上面的

  • klass: 'java/util/stream/FindOps$$Lambda$47’信息
    搜索了下看有小伙伴说是jdk1.8.0.25 lambada的一个bug升级最新1.8版本可解决,然后升级了下,是暂时还没有挂掉,但是想请教大牛怎么确认下这个问题,
    最好有详细的思路和具体的工具使用方法(小白没啥经验,看到一些分析的文章总是被搁浅到怎么使用上,都是直接略过拿到了结果,可能对小白来说最重要的还是怎么使用,怎么定位。。)
    这里也希望大牛能给一下经验,需要积累学习那些(路径?)才能对类似问题有思路(这里能很多东西原来压根就不知道,就很难有思路去排查,确认),万分感谢!
4397 阅读
请先登录,再评论

该评论已被删除

2年前

从crash日志来看和lambda有关,可以把crash日志补全看看吗

12年前
2年前回复

还有,这里挂掉时也生成了core,但是这个core用什么来查看,怎么分析呢。😰

2年前