性能文章>一次诡异的JVM堆外内存泄漏>

一次诡异的JVM堆外内存泄漏转载

6月前
236004

导语

堆外metaspace内存占用高达3GB多 ,机器内存耗尽后导致宕机,jmap查看JVM对象信息,发现大量和反射相关对象被生成,本篇介绍的是一次诡异的jvm对外内存泄漏排查及优化过程,希望对阅读的各位有所帮助。

正文

一、现象

  • 报警详情: MEM usage above 90% (current value: 0.9731329333728482) 
  • 堆外metaspace内存占用高达3GB多 
  • 机器内存耗尽,宕机

二、概念

元空间是jdk1.8开始取代永久代的内存模型,被jvm使用受操作系统管辖的直接内存区域。

jdk1.7内存结构:

一次诡异的JVM堆外内存泄漏数据图表-heapdump性能社区


jdk1.8内存结构:

一次诡异的JVM堆外内存泄漏数据图表-heapdump性能社区


三、排查思路

  •  metaspace增长是逐渐增多,增长速度不均匀,考虑是接口调用或mq 
  •  metaspace主要存放类信息,所以主要怀疑:动态类生成类库的使用 
  •  项目中使用的第三方库:fastjson,dubbo,aspectj,spring 】 首先排除了fastjson,dubbo,spring,因为足够稳定和靠谱 
  • 但是预发环境和测试环境,依旧无法重现问题

四、排查过程

  • jmap查看JVM对象信息,发现大量和反射相关对象被生成
admin@ip-xxx:~/logs/xxx$ jmap -histo pid

 num     #instances         #bytes  class name
----------------------------------------------
   1:        836613      102185264  [C
   2:        128213       30887816  [B
   3:        546707       13120968  java.lang.String
   4:        390652       12500864  java.util.HashMap$Node
   5:         58502       10908984  [I
   6:        121161       10662168  java.lang.reflect.Method
   7:        149994       10117912  [Ljava.lang.Object;
   8:         61139        6432184  [Ljava.util.HashMap$Node;
   9:        160882        5148224  java.util.concurrent.ConcurrentHashMap$Node
  10:        172682        3822600  [Ljava.lang.Class;
  11:         58938        2829024  java.util.HashMap
  12:         24263        2688136  java.lang.Class
  13:        110260        2646240  java.util.ArrayList
  14:         65896        2635840  java.util.LinkedHashMap$Entry
  15:         41654        2252544  [Ljava.lang.String;
  16:          2459        1894392  [Ljava.util.concurrent.ConcurrentHashMap$Node;
  17:         24752        1782144  java.lang.reflect.Field
  18:         73982        1775568  java.lang.StringBuilder
  19:         29437        1648472  java.util.LinkedHashMap
  20:         51507        1648224  java.lang.ref.WeakReference
  21:         34020        1632960  org.aspectj.weaver.reflect.ShadowMatchImpl
  22:         30742        1229680  com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl$1
  23:         76420        1222720  java.lang.Integer
  24:         34020        1088640  org.aspectj.weaver.patterns.ExposedState
  25:         12658        1012640  java.lang.reflect.Constructor
  26:         24377         975080  java.util.HashMap$KeyIterator
  27:         59090         945440  java.lang.Object
  28:         23290         931600  java.lang.ref.SoftReference
  29:         18518         919184  [Ljava.lang.reflect.Method;
  30:         35328         847872  com.dianping.cat.internal.shaded.io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry
  31:         24738         791616  com.sun.xml.internal.stream.events.CharacterEvent
  32:         13612         762272  sun.nio.cs.UTF_8$Encoder
  33:         31471         755304  io.termd.core.term.Feature
  34:         23095         739040  com.sun.org.apache.xerces.internal.xni.QName
  35:         10412         666368  java.util.regex.Matcher
  36:         11898         666288  java.beans.MethodDescriptor


●  增加jvm启动参数:-XX:+TraceClassLoading -XX:+TraceClassUnloading 
●  查看项目的类加载日志

[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_InboxBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_RefundBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundLineBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_PushBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundPushArgs from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_3_PushRefundLine from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_InboxBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_PushBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_PushArgs from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_RefundBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundLineBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_PushBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundPushArgs from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_3_PushRefundLine from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_InboxBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_RefundBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundLineBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_PushBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]

 

xxx 03:29:32.045 [http-nio-8888-exec-7] INFO   xxx.proxy.push.PushProxy:[80] - SendMixPush, url: xxx, body: {"message":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx},"sns":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx}}
xxx 03:29:48.813 [http-nio-8888-exec-5] INFO   xxx.proxy.push.PushProxy:[80] - SendMixPush, url: xxx, body: {"message":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx},"sns":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx}}
xxx 03:29:56.844 [http-nio-8888-exec-2] INFO   xxx.proxy.push.PushProxy:[80] - SendMixPush, url: xxx, body: {"message":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx},"sns":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx}}
xxx 03:30:03.787 [http-nio-8888-exec-9] INFO   xxx.proxy.push.PushProxy:[80] - SendMixPush, url: xxx, body: {"message":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx},"sns":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx}}

 

  • 查看具体代码
public static String toSneakJSONString(Object object){
      SerializeConfig serializeConfig = new SerializeConfig();
      serializeConfig.setPropertyNamingStrategy(PropertyNamingStrategy.SnakeCase);
      return JSONObject.toJSONString(object, serializeConfig, SerializerFeature.DisableCircularReferenceDetect);
  }

 

  • 《fastjson 官方建议 》https://github.com/alibaba/fastjson/issues/385#issuecomment-209281348

五、问题根源和解决方案

1.问题根源分析

SerializeConfig 默认会激活 a**,在序列化对象时会为对象生成代理类,然后通过执行代理进行序列化操作,通过这样优化来提高执行性能,但在应用不合理每次新创建 config 的时候就会导致大量生成代码类反而拖慢性能。反序列化时的 ParserConfig 也是同理。

在 jdk8 之前这些代理类会充满 Perm 区导致 FullGC,浪费点 CPU 也不会有大问题,但在 JDK8 中,这些类会大量创建直至充满物理机内存,操作系统检测到该进程是危险进程,出于自我保护机制,进而导致进程被系统杀掉。

2.解决方案

  •  代码维度,将SerializeConfig缓存起来,或者使用SerializeConfig.globalInstance全局配置,避免请求维度动态生成类并加载到元空间
  • 调整jvm启动参数,限制metaspace最大内存-XX:MaxMetaspaceSize,宁可FGC或者OOM也不要进程被杀

参考

fastjosn官方issues1 -- https://github.com/alibaba/fastjson/issues/2109

fastjson官方issues2 -- https://github.com/alibaba/fastjson/issues/385

什么是元空间Metaspace -- https://www.infoq.cn/article/Java-PERMGEN-Removed

fastjson中用到的ASM -- http://pwn4.fun/2017/02/07/JAVA-ASM%E5%AD%97%E8%8A%82%E7%A0%81%E6%A1%86%E6%9E%B6/

a**官网 -- https://a**.ow2.io/

 

更多思考

JVM的泄漏及优化是我们老生常谈的问题,更多关于JVM的知识大家可以阅读以下内容加深阅读

一次 JVM 进程退出分析

一次jvm调优实战

点赞收藏
嘿小黑
请先登录,感受更多精彩内容
快去登录吧,你将获得
  • 浏览更多精彩评论
  • 和开发者讨论交流,共同进步

为你推荐

【全网首发】一次想不到的 Bootstrap 类加载器带来的 Native 内存泄露分析

【全网首发】一次想不到的 Bootstrap 类加载器带来的 Native 内存泄露分析

4
0