HeapDump性能社区

性能工具
登录注册
性能文章>一次诡异的JVM堆外内存泄漏>

一次诡异的JVM堆外内存泄漏转载

嘿小黑
2年前
492004

导语

堆外metaspace内存占用高达3GB多 ，机器内存耗尽后导致宕机，jmap查看JVM对象信息,发现大量和反射相关对象被生成，本篇介绍的是一次诡异的jvm对外内存泄漏排查及优化过程，希望对阅读的各位有所帮助。

正文

一、现象

  • 报警详情: MEM usage above 90% (current value: 0.9731329333728482) 
  • 堆外metaspace内存占用高达3GB多 
  • 机器内存耗尽，宕机

二、概念

元空间是jdk1.8开始取代永久代的内存模型,被jvm使用受操作系统管辖的直接内存区域。

jdk1.7内存结构：

一次诡异的JVM堆外内存泄漏数据图表-heapdump性能社区


jdk1.8内存结构:

一次诡异的JVM堆外内存泄漏数据图表-heapdump性能社区


三、排查思路

  •  metaspace增长是逐渐增多，增长速度不均匀，考虑是接口调用或mq 
  •  metaspace主要存放类信息，所以主要怀疑:动态类生成类库的使用 
  •  项目中使用的第三方库：fastjson,dubbo,aspectj,spring 】 首先排除了fastjson,dubbo,spring，因为足够稳定和靠谱 
  • 但是预发环境和测试环境，依旧无法重现问题

四、排查过程

  • jmap查看JVM对象信息,发现大量和反射相关对象被生成
admin@ip-xxx:~/logs/xxx$ jmap -histo pid

 num     #instances         #bytes  class name
----------------------------------------------
   1:        836613      102185264  [C
   2:        128213       30887816  [B
   3:        546707       13120968  java.lang.String
   4:        390652       12500864  java.util.HashMap$Node
   5:         58502       10908984  [I
   6:        121161       10662168  java.lang.reflect.Method
   7:        149994       10117912  [Ljava.lang.Object;
   8:         61139        6432184  [Ljava.util.HashMap$Node;
   9:        160882        5148224  java.util.concurrent.ConcurrentHashMap$Node
  10:        172682        3822600  [Ljava.lang.Class;
  11:         58938        2829024  java.util.HashMap
  12:         24263        2688136  java.lang.Class
  13:        110260        2646240  java.util.ArrayList
  14:         65896        2635840  java.util.LinkedHashMap$Entry
  15:         41654        2252544  [Ljava.lang.String;
  16:          2459        1894392  [Ljava.util.concurrent.ConcurrentHashMap$Node;
  17:         24752        1782144  java.lang.reflect.Field
  18:         73982        1775568  java.lang.StringBuilder
  19:         29437        1648472  java.util.LinkedHashMap
  20:         51507        1648224  java.lang.ref.WeakReference
  21:         34020        1632960  org.aspectj.weaver.reflect.ShadowMatchImpl
  22:         30742        1229680  com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl$1
  23:         76420        1222720  java.lang.Integer
  24:         34020        1088640  org.aspectj.weaver.patterns.ExposedState
  25:         12658        1012640  java.lang.reflect.Constructor
  26:         24377         975080  java.util.HashMap$KeyIterator
  27:         59090         945440  java.lang.Object
  28:         23290         931600  java.lang.ref.SoftReference
  29:         18518         919184  [Ljava.lang.reflect.Method;
  30:         35328         847872  com.dianping.cat.internal.shaded.io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry
  31:         24738         791616  com.sun.xml.internal.stream.events.CharacterEvent
  32:         13612         762272  sun.nio.cs.UTF_8$Encoder
  33:         31471         755304  io.termd.core.term.Feature
  34:         23095         739040  com.sun.org.apache.xerces.internal.xni.QName
  35:         10412         666368  java.util.regex.Matcher
  36:         11898         666288  java.beans.MethodDescriptor


●  增加jvm启动参数：-XX:+TraceClassLoading -XX:+TraceClassUnloading 
●  查看项目的类加载日志

[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_InboxBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_RefundBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundLineBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_PushBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundPushArgs from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_3_PushRefundLine from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_InboxBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_PushBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_PushArgs from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_RefundBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundLineBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_PushBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundPushArgs from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_3_PushRefundLine from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_InboxBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_RefundBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundLineBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_PushBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]

 

xxx 03:29:32.045 [http-nio-8888-exec-7] INFO   xxx.proxy.push.PushProxy:[80] - SendMixPush, url: xxx, body: {"message":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx},"sns":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx}}
xxx 03:29:48.813 [http-nio-8888-exec-5] INFO   xxx.proxy.push.PushProxy:[80] - SendMixPush, url: xxx, body: {"message":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx},"sns":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx}}
xxx 03:29:56.844 [http-nio-8888-exec-2] INFO   xxx.proxy.push.PushProxy:[80] - SendMixPush, url: xxx, body: {"message":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx},"sns":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx}}
xxx 03:30:03.787 [http-nio-8888-exec-9] INFO   xxx.proxy.push.PushProxy:[80] - SendMixPush, url: xxx, body: {"message":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx},"sns":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx}}

 

  • 查看具体代码
public static String toSneakJSONString(Object object){
      SerializeConfig serializeConfig = new SerializeConfig();
      serializeConfig.setPropertyNamingStrategy(PropertyNamingStrategy.SnakeCase);
      return JSONObject.toJSONString(object, serializeConfig, SerializerFeature.DisableCircularReferenceDetect);
  }

 

  • 《fastjson 官方建议 》https://github.com/alibaba/fastjson/issues/385#issuecomment-209281348

五、问题根源和解决方案

1.问题根源分析

SerializeConfig 默认会激活 a**，在序列化对象时会为对象生成代理类，然后通过执行代理进行序列化操作，通过这样优化来提高执行性能，但在应用不合理每次新创建 config 的时候就会导致大量生成代码类反而拖慢性能。反序列化时的 ParserConfig 也是同理。

在 jdk8 之前这些代理类会充满 Perm 区导致 FullGC，浪费点 CPU 也不会有大问题，但在 JDK8 中，这些类会大量创建直至充满物理机内存，操作系统检测到该进程是危险进程,出于自我保护机制,进而导致进程被系统杀掉。

2.解决方案

  •  代码维度,将SerializeConfig缓存起来,或者使用SerializeConfig.globalInstance全局配置,避免请求维度动态生成类并加载到元空间
  • 调整jvm启动参数,限制metaspace最大内存-XX:MaxMetaspaceSize,宁可FGC或者OOM也不要进程被杀

参考

fastjosn官方issues1 -- https://github.com/alibaba/fastjson/issues/2109

fastjson官方issues2 -- https://github.com/alibaba/fastjson/issues/385

什么是元空间Metaspace -- https://www.infoq.cn/article/Java-PERMGEN-Removed

fastjson中用到的ASM -- http://pwn4.fun/2017/02/07/JAVA-ASM%E5%AD%97%E8%8A%82%E7%A0%81%E6%A1%86%E6%9E%B6/

a**官网 -- https://a**.ow2.io/

 

更多思考

JVM的泄漏及优化是我们老生常谈的问题，更多关于JVM的知识大家可以阅读以下内容加深阅读

一次 JVM 进程退出分析

一次jvm调优实战

原文链接：https://mp.weixin.qq.com/s/_bZeMmpNksiuzc4-PwC8lA

点赞收藏
嘿小黑
请先登录，感受更多精彩内容
快去登录吧，你将获得
  • 浏览更多精彩评论
  • 和开发者讨论交流，共同进步
4
0
Lv2
嘿小黑

徽章

推荐阅读

一次 GDB 源码角度分析 jvm 无响应问题Vue混淆与还原两个例子带你入门 Disruptor13 轻量级锁的重入 以及 线程1获取轻量级锁并释放线程2获取锁 的调试朋友们，就在今天，JDK 21，它终于带着重磅新特性正式发布了！！！为什么 JVM 叫做基于栈的 RISC 虚拟机
站内搜索
Linux
fgc
字节码
堆外内存
火焰图
堆外内存泄漏
java
jvm
gc
linux
内存泄漏
内核

本月精选性能专题

本月精选线上案例

本月精选原创好文

HeapDump

有性能问题，上HeapDump性能社区

Copyright © 2019-2024 heapdump.cn 版权所有 浙ICP备17054444号

联系我们

HeapDump性能社区

网媒渠道

堆堆酱

扫码加堆堆酱好友

友情链接

PerfMa官网SegmentFault思否TesterHomeXPocket开源社区DCloud 问答社区稀土掘金博客园W3CTech