性能文章>一次full gc的排查经历>

一次full gc的排查经历原创

8966111

问题描述

1月22日准备发布设备指纹新版本迭代过程中,在预发环境进行采集请求的压测发现了个现象,监控中的RT存在规律的尖点。

接着又尝试master代码,发现不存在这个问题,那基本确认是新代码引入的问题了。

问题定位

猜测是full gc导致RT变长,用jstat命令查看果然不出所料。

image.png

那么为什么会出现频繁的full gc呢?接下来开始漫长的排查过程:

代码定位

由于新版本改动点比较多,所以我们尝试一步步还原代码压测,最终发现是kryo包从2.16升级到2.24.0导致了这个现象。

堆内存排查

full gc,第一反应都会以为是由于堆中老年代对象过多引起的,所以二话不说开始执行jmap dump堆内存,结果4G多的内存下载下来分析后并无头绪。

jfr分析排查

执行jmc命令记录jfr文件进行分析,结果发现在full gc前后metaspace明显减少了(项目使用的是java1.8):

image.png

image.png

并且在类卸载那一栏中有非常多的sun.reflect.GeneratedSerializationConstructorAccessor被卸载:

那么应该是元空间中频繁加载了过多动态类导致的full gc。再次用jstat命令进行确认了一遍,发现在每一次full gc后metaspace都有明显的变少,问题基本确认。

问题深入

那么kryo包的这2个版本有什么区别呢?为什么会频繁使用动态类加载?又开始了漫长的分析过程:

kryo是什么,怎么用

Kryo 是一个快速序列化/反序列化工具,其使用了字节码生成机制(底层依赖了ASM库),因此具有比较好的运行速度。

项目中使用kryo序列化代码:

public byte[] kryoSerialize(AerospikeItemNewSession aerospikeItemNewSession) {

        byte saveResultBytes[] = null;

        ByteArrayOutputStream byt1 = new ByteArrayOutputStream();

        Output output = new Output(byt1);

        try {

            Kryo kryo = new Kryo();

            kryo.setReferences(false);//是否需要支持循环引用

            kryo.setRegistrationRequired(false);//是否开启注册行为

            kryo.setInstantiatorStrategy(new StdInstantiatorStrategy());//反序列化策略

            kryo.register(AerospikeItemNewSession.class);

            kryo.writeObject(output, aerospikeItemNewSession);

            output.flush();

            saveResultBytes = byt1.toByteArray();

        } catch (Throwable ex) {

            logger.error("kryoSerialize occur,");

        } finally {

            try {

                byt1.flush();

                byt1.close();

                output.close();

            } catch (IOException e) {

                logger.error("kryoSerialize close object occur,", e);

            }

        }

        return saveResultBytes;

    }

项目中使用kryo反序列化代码:

public String kryoDeserialize(byte[] saveResultBytes) {

        String result = "";

        AerospikeItemNewSession aerospikeItemNewSession;

        ByteArrayInputStream bis1 = new ByteArrayInputStream(saveResultBytes);

        Input input = new Input(bis1);

        try {

            Kryo kryo = new Kryo();

            kryo.setReferences(false);//是否需要支持循环引用

            kryo.setRegistrationRequired(false);//是否开启注册行为

            kryo.setInstantiatorStrategy(new StdInstantiatorStrategy());//反序列化策略

            kryo.register(AerospikeItemNewSession.class);

            if ((aerospikeItemNewSession = kryo.readObject(input, AerospikeItemNewSession.class)) != null) {

                if (aerospikeItemNewSession.isCompressed()) {

                    result = CompressUtil.ungzip(aerospikeItemNewSession.getValue());

                } else {

                    result = new String(aerospikeItemNewSession.getValue(), "utf-8");

                }

            }

        } catch (Exception ex) {

            logger.error("kryoDeserialize occur,", ex);

        } finally {

            try {

                bis1.close();

                input.close();

            } catch (IOException e) {

                logger.error("kryoDeserialize close object occur,", e);

            }

        }

        return result;

    }

进入误区

因为是设备指纹的采集请求,所以想当然认为是kryo.writeObject(output, aerospikeItemNewSession);序列化代码出现问题。

接着开始进行源码研究和调试,发现并没有出现动态类创建的方法和代码,后面又进行了如下尝试:

  • 启动时用class.forName将AerospikeItemNewSession类装载一次, 无用

  • 将AerospikeItemNewSession类改成静态类,无用

  • 将AerospikeItemNewSession外部public类(原先是写在同级类下,只有同个包访问权限),无用

  • 将类使用到的序列化器FieldSerializer改成2.16版本的代码,无用

地毯式搜索

既然是反射动态生成类引起的问题,所以干脆在kryo代码中全局搜索reflect相关的代码,发现kyro.java下的一段代码有点可疑:

public ObjectInstantiator newInstantiatorOf (final Class type) {

            if (!Util.isAndroid) {

                // Use ReflectASM if the class is not a non-static member class.

                Class enclosingType = type.getEnclosingClass();

                boolean isNonStaticMemberClass = enclosingType != null && type.isMemberClass()

                    && !Modifier.isStatic(type.getModifiers());

                if (!isNonStaticMemberClass) {

                    try {

                        final ConstructorAccess access = ConstructorAccess.get(type);

                        return new ObjectInstantiator() {

                            public Object newInstance () {

                                try {

                                    return access.newInstance();

                                } catch (Exception ex) {

                                    throw new KryoException("Error constructing instance of class: " + className(type), ex);

                                }

                            }

                        };

                    } catch (Exception ignored) {

                    }

                }

            }

            // Reflection.

            try {

                Constructor ctor;

                try {

                    ctor = type.getConstructor((Class[])null);

                } catch (Exception ex) {

                    ctor = type.getDeclaredConstructor((Class[])null);

                    ctor.setAccessible(true);

                }

                final Constructor constructor = ctor;

                return new ObjectInstantiator() {

                    public Object newInstance () {

                        try {

                            return constructor.newInstance();

                        } catch (Exception ex) {

                            throw new KryoException("Error constructing instance of class: " + className(type), ex);

                        }

                    }

                };

            } catch (Exception ignored) {

            }

            if (fallbackStrategy == null) {

                if (type.isMemberClass() && !Modifier.isStatic(type.getModifiers()))

                    throw new KryoException("Class cannot be created (non-static member class): " + className(type));

                else

                    throw new KryoException("Class cannot be created (missing no-arg constructor): " + className(type));

            }

            // InstantiatorStrategy.

            return fallbackStrategy.newInstantiatorOf(type);

        }

接着往前寻找他的调用栈,发现这是在反序列化的时候使用的,用于生成AerospikeItemNewSession实例,突然意识到是不是反序列也需要反射生成实例呢,而采集请求也调用到了这块,马上去对比了下2.16的代码,发现protected ObjectInstantiator newInstantiator (final Class type)方法有差别:

2.16的代码是这样的:

/** Returns a new instantiator for creating new instances of the specified type. By default, an instantiator is returned that

 * uses reflection if the class has a zero argument constructor, an exception is thrown. If a

 * {@link #setInstantiatorStrategy(InstantiatorStrategy) strategy} is set, it will be used instead of throwing an exception. */

protected ObjectInstantiator newInstantiator (final Class type) {

    if (!Util.isAndroid) {

        // ReflectASM.

        try {

            final ConstructorAccess access = ConstructorAccess.get(type);

            return new ObjectInstantiator() {

                public Object newInstance () {

                    try {

                        return access.newInstance();

                    } catch (Exception ex) {

                        throw new KryoException("Error constructing instance of class: " + className(type), ex);

                    }

                }

            };

        } catch (Exception ignored) {

        }

    }

    // Reflection.

    try {

        Constructor ctor;

        try {

            ctor = type.getConstructor((Class[])null);

        } catch (Exception ex) {

            ctor = type.getDeclaredConstructor((Class[])null);

            ctor.setAccessible(true);

        }

        final Constructor constructor = ctor;

        return new ObjectInstantiator() {

            public Object newInstance () {

                try {

                    return constructor.newInstance();

                } catch (Exception ex) {

                    throw new KryoException("Error constructing instance of class: " + className(type), ex);

                }

            }

        };

    } catch (Exception ignored) {

    }

    if (strategy == null) {

        if (type.isMemberClass() && !Modifier.isStatic(type.getModifiers()))

            throw new KryoException("Class cannot be created (non-static member class): " + className(type));

        else

            throw new KryoException("Class cannot be created (missing no-arg constructor): " + className(type));

    }

    // InstantiatorStrategy.

    return strategy.newInstantiatorOf(type);

}

2.24.0的代码是这样的:

/** Returns a new instantiator for creating new instances of the specified type. By default, an instantiator is returned that

 * uses reflection if the class has a zero argument constructor, an exception is thrown. If a

 * {@link #setInstantiatorStrategy(InstantiatorStrategy) strategy} is set, it will be used instead of throwing an exception. */

protected ObjectInstantiator newInstantiator (final Class type) {

    // InstantiatorStrategy.

    return strategy.newInstantiatorOf(type);

}

2.24.0里把2.16里return strategy.newInstantiatorOf(type);上面的很大一段代码挪到了DefaultInstantiatorStrategy的实现里了,而项目中显式设置了StdInstantiatorStrategy策略,因此2.24.0就不会执行那段代码而直接采用StdInstantiatorStrategy创建实例化器,2.16会执行上面那段代码创建实例化器。为了证实我的猜想,分别调试了2个版本的代码,发现确实如此。

2.16版本反序列调用链路:

Kryo.readObject (Input input, Class<T> type) -> FieldSerializer.read (Kryo kryo, Input input, Class<T> type) -> FieldSerializer.create (Kryo kryo, Input input, Class<T> type) -> Kryo.newInstance (Class<T> type) -> Kryo.newInstantiator (final Class type)

2.24版本反序列调用链路:

Kryo.readObject (Input input, Class<T> type) -> FieldSerializer.read (Kryo kryo, Input input, Class<T> type) -> FieldSerializer.create (Kryo kryo, Input input, Class<T> type) -> Kryo.newInstance (Class<T> type) -> Kryo.newInstantiator (final Class type) -> StdInstantiatorStrategy.newInstantiatorOf(Class type)

问题解决

基于上面的分析思路,去掉策略设置kryo.setInstantiatorStrategy(new StdInstantiatorStrategy());,默认就会使用DefaultInstantiatorStrategy执行,执行的逻辑和2.16保持一致,再进行一次压测之后不再出现full gc。

kryo源码分析

为什么执行了StdInstantiatorStrategy会造成频繁创建class?

静态class定义创建和加载

从上面的分析可以看出2.16中Kryo.newInstantiator (final Class type)方法和2.24中DefaultInstantiatorStrategy类会优先执行一段如下的代码逻辑:

if (!Util.isAndroid) {

    // Use ReflectASM if the class is not a non-static member class.

    Class enclosingType = type.getEnclosingClass();

    boolean isNonStaticMemberClass = enclosingType != null && type.isMemberClass()

        && !Modifier.isStatic(type.getModifiers());

    if (!isNonStaticMemberClass) {

        try {

            final ConstructorAccess access = ConstructorAccess.get(type);

            return new ObjectInstantiator() {

                public Object newInstance () {

                    try {

                        return access.newInstance();

                    } catch (Exception ex) {

                        throw new KryoException("Error constructing instance of class: " + className(type), ex);

                    }

                }

            };

        } catch (Exception ignored) {

        }

    }

}

这段代码返回了ObjectInstantiator类型,里面newInstatnce方法就是用于创建反序列化类。代码里ConstructorAccess类是一个反序列化实例的构造器访问类,可以理解为一个能够构造实例的工厂类(和ObjectInstantiator有点像),那么这个方法final ConstructorAccess access = ConstructorAccess.get(type);是怎么获取和构造这个工厂呢?进入get方法查看源码:

public static <T> ConstructorAccess<T> get(Class<T> type) {

    Class enclosingType = type.getEnclosingClass();

    boolean isNonStaticMemberClass = enclosingType != null && type.isMemberClass() && !Modifier.isStatic(type.getModifiers());

    String className = type.getName();

    String accessClassName = className + "ConstructorAccess";

    if (accessClassName.startsWith("java.")) {

        accessClassName = "reflectasm." + accessClassName;

    }

 

    Class accessClass = null;

    AccessClassLoader loader = AccessClassLoader.get(type);

    synchronized(loader) {

        try {

            accessClass = loader.loadClass(accessClassName);

        } catch (ClassNotFoundException var18) {

            String accessClassNameInternal = accessClassName.replace('.', '/');

            String classNameInternal = className.replace('.', '/');

            boolean isPrivate = false;

            String enclosingClassNameInternal;

            Constructor constructor;

            if (!isNonStaticMemberClass) {

                enclosingClassNameInternal = null;

 

                try {

                    constructor = type.getDeclaredConstructor((Class[])null);

                    isPrivate = Modifier.isPrivate(constructor.getModifiers());

                } catch (Exception var17) {

                    throw new RuntimeException("Class cannot be created (missing no-arg constructor): " + type.getName(), var17);

                }

 

                if (isPrivate) {

                    throw new RuntimeException("Class cannot be created (the no-arg constructor is private): " + type.getName());

                }

            } else {

                enclosingClassNameInternal = enclosingType.getName().replace('.', '/');

 

                try {

                    constructor = type.getDeclaredConstructor(enclosingType);

                    isPrivate = Modifier.isPrivate(constructor.getModifiers());

                } catch (Exception var16) {

                    throw new RuntimeException("Non-static member class cannot be created (missing enclosing class constructor): " + type.getName(), var16);

                }

 

                if (isPrivate) {

                    throw new RuntimeException("Non-static member class cannot be created (the enclosing class constructor is private): " + type.getName());

                }

            }

 

            ClassWriter cw = new ClassWriter(0);

            cw.visit(196653, 33, accessClassNameInternal, (String)null, "com/esotericsoftware/reflectasm/ConstructorAccess", (String[])null);

            insertConstructor(cw);

            insertNewInstance(cw, classNameInternal);

            insertNewInstanceInner(cw, classNameInternal, enclosingClassNameInternal);

            cw.visitEnd();

            accessClass = loader.defineClass(accessClassName, cw.toByteArray());

        }

    }

 

    try {

        ConstructorAccess<T> access = (ConstructorAccess)accessClass.newInstance();

        access.isNonStaticMemberClass = isNonStaticMemberClass;

        return access;

    } catch (Exception var15) {

        throw new RuntimeException("Error constructing constructor access class: " + accessClassName, var15);

    }

}

可以看到这段代码主要采用的是classloader方式defineClass和loadClass,accessClassName是固定的(className+“ConstructorAccess”),第一次调用accessClass = loader.loadClass(accessClassName);的地方会抛出ClassNotFoundException异常,异常捕获后开始创建accessClass,只要被序列化的类不是私有的内部类,则可以通过这种方式创建出工厂类accessClass,然后accessClass = loader.defineClass(accessClassName, cw.toByteArray());定义到元空间中,以后loader.loadClass(accessClassName)的调用则加载同一个类定义。

动态类创建

2.16中Kryo.newInstantiator (final Class type)方法和2.24中DefaultInstantiatorStrategy类的最后会调用strategy去动态生成class定义,也就是说只有当静态class创建加载失败的时候,才会执行策略,接下来我们看下StdInstantiatorStrategy.java的实现(以下是2.24.0的代码,2.16也是类似):

public <T> ObjectInstantiator<T> newInstantiatorOf(Class<T> type) {

 

    if(PlatformDescription.isThisJVM(SUN) || PlatformDescription.isThisJVM(OPENJDK)) {

       // The UnsafeFactoryInstantiator would also work. But according to benchmarks, it is 2.5

       // times slower. So I prefer to use this one

       return new SunReflectionFactoryInstantiator<T>(type);

    }

    else if(PlatformDescription.isThisJVM(JROCKIT)) {

       if(VM_VERSION.startsWith("1.4")) {

          // JRockit vendor version will be RXX where XX is the version

          // Versions prior to 26 need special handling

          // From R26 on, java.vm.version starts with R

          if(!VENDOR_VERSION.startsWith("R")) {

             // On R25.1 and R25.2, ReflectionFactory should work. Otherwise, we must use the

             // Legacy instantiator.

             if(VM_INFO == null || !VM_INFO.startsWith("R25.1") || !VM_INFO.startsWith("R25.2")) {

                return new JRockitLegacyInstantiator<T>(type);

             }

          }

       }

       // After that, JRockit became compliant with HotSpot

       return new SunReflectionFactoryInstantiator<T>(type);

    }

    else if(PlatformDescription.isThisJVM(DALVIK)) {

       if(ANDROID_VERSION <= 10) {

          // Android 2.3 Gingerbread and lower

          return new Android10Instantiator<T>(type);

       }

       if(ANDROID_VERSION <= 17) {

          // Android 3.0 Honeycomb to 4.2 Jelly Bean

          return new Android17Instantiator<T>(type);

       }

       // Android 4.3 and higher (hopefully)

       return new Android18Instantiator<T>(type);

    }

    else if(PlatformDescription.isThisJVM(GNU)) {

       return new GCJInstantiator<T>(type);

    }

    else if(PlatformDescription.isThisJVM(PERC)) {

       return new PercInstantiator<T>(type);

    }

 

    // Fallback instantiator, should work with most modern JVM

    return new UnsafeFactoryInstantiator<T>(type);

 

 }

可以看出如果是sun的jdk会执行return new SunReflectionFactoryInstantiator<T>(type);,SunReflectionFactoryInstantiator就是ObjectInstantiator的实现类,内部维护了一个private final Constructor mungedConstructor;属性,这是一个构造函数类实例,可以理解为和5.1中的ConstructorAccess工厂类类似,能够实例化反序列化类。mungedConstructor属性最终是被ReflectionFactory.generateConstructor生成:

private final Constructor<?> generateConstructor(Class<?> var1, Constructor<?> var2) {

    SerializationConstructorAccessorImpl var3 = (new MethodAccessorGenerator()).generateSerializationConstructor(var1, var2.getParameterTypes(), var2.getExceptionTypes(), var2.getModifiers(), var2.getDeclaringClass());

    Constructor var4 = this.newConstructor(var2.getDeclaringClass(), var2.getParameterTypes(), var2.getExceptionTypes(), var2.getModifiers(), langReflectAccess().getConstructorSlot(var2), langReflectAccess().getConstructorSignature(var2), langReflectAccess().getConstructorAnnotations(var2), langReflectAccess().getConstructorParameterAnnotations(var2));

    this.setConstructorAccessor(var4, var3);

    var4.setAccessible(true);

    return var4;

}

而mungedConstructor中的constructorAccessor(Constructor中的newInstance方法都会调用constructorAccessor.newInstatnce)是由MethodAccessorGenerator.generate方法生成,代码如下:

private MagicAccessorImpl generate(final Class<?> var1, String var2, Class<?>[] var3, Class<?> var4, Class<?>[] var5, int var6, boolean var7, boolean var8, Class<?> var9) {

    ByteVector var10 = ByteVectorFactory.create();

    this.asm = new ClassFileAssembler(var10);

    this.declaringClass = var1;

    this.parameterTypes = var3;

    this.returnType = var4;

    this.modifiers = var6;

    this.isConstructor = var7;

    this.forSerialization = var8;

    this.asm.emitMagicAndVersion();

    short var11 = 42;

    boolean var12 = this.usesPrimitiveTypes();

    if (var12) {

        var11 = (short)(var11 + 72);

    }

 

    if (var8) {

        var11 = (short)(var11 + 2);

    }

 

    var11 += (short)(2 * this.numNonPrimitiveParameterTypes());

    this.asm.emitShort(add(var11, (short)1));

    final String var13 = generateName(var7, var8);

    this.asm.emitConstantPoolUTF8(var13);

    this.asm.emitConstantPoolClass(this.asm.cpi());

    this.thisClass = this.asm.cpi();

    if (var7) {

        if (var8) {

            this.asm.emitConstantPoolUTF8("sun/reflect/SerializationConstructorAccessorImpl");

        } else {

            this.asm.emitConstantPoolUTF8("sun/reflect/ConstructorAccessorImpl");

        }

    } else {

        this.asm.emitConstantPoolUTF8("sun/reflect/MethodAccessorImpl");

    }

 

    this.asm.emitConstantPoolClass(this.asm.cpi());

    this.superClass = this.asm.cpi();

    this.asm.emitConstantPoolUTF8(getClassName(var1, false));

    this.asm.emitConstantPoolClass(this.asm.cpi());

    this.targetClass = this.asm.cpi();

    short var14 = 0;

    if (var8) {

        this.asm.emitConstantPoolUTF8(getClassName(var9, false));

        this.asm.emitConstantPoolClass(this.asm.cpi());

        var14 = this.asm.cpi();

    }

 

    this.asm.emitConstantPoolUTF8(var2);

    this.asm.emitConstantPoolUTF8(this.buildInternalSignature());

    this.asm.emitConstantPoolNameAndType(sub(this.asm.cpi(), (short)1), this.asm.cpi());

    if (this.isInterface()) {

        this.asm.emitConstantPoolInterfaceMethodref(this.targetClass, this.asm.cpi());

    } else if (var8) {

        this.asm.emitConstantPoolMethodref(var14, this.asm.cpi());

    } else {

        this.asm.emitConstantPoolMethodref(this.targetClass, this.asm.cpi());

    }

 

    this.targetMethodRef = this.asm.cpi();

    if (var7) {

        this.asm.emitConstantPoolUTF8("newInstance");

    } else {

        this.asm.emitConstantPoolUTF8("invoke");

    }

 

    this.invokeIdx = this.asm.cpi();

    if (var7) {

        this.asm.emitConstantPoolUTF8("([Ljava/lang/Object;)Ljava/lang/Object;");

    } else {

        this.asm.emitConstantPoolUTF8("(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;");

    }

 

    this.invokeDescriptorIdx = this.asm.cpi();

    this.nonPrimitiveParametersBaseIdx = add(this.asm.cpi(), (short)2);

 

    for(int var15 = 0; var15 < var3.length; ++var15) {

        Class var16 = var3[var15];

        if (!isPrimitive(var16)) {

            this.asm.emitConstantPoolUTF8(getClassName(var16, false));

            this.asm.emitConstantPoolClass(this.asm.cpi());

        }

    }

 

    this.emitCommonConstantPoolEntries();

    if (var12) {

        this.emitBoxingContantPoolEntries();

    }

 

    if (this.asm.cpi() != var11) {

        throw new InternalError("Adjust this code (cpi = " + this.asm.cpi() + ", numCPEntries = " + var11 + ")");

    } else {

        this.asm.emitShort((short)1);

        this.asm.emitShort(this.thisClass);

        this.asm.emitShort(this.superClass);

        this.asm.emitShort((short)0);

        this.asm.emitShort((short)0);

        this.asm.emitShort((short)2);

        this.emitConstructor();

        this.emitInvoke();

        this.asm.emitShort((short)0);

        var10.trim();

        final byte[] var17 = var10.getData();

        return (MagicAccessorImpl)AccessController.doPrivileged(new PrivilegedAction<MagicAccessorImpl>() {

            public MagicAccessorImpl run() {

                try {

                    return (MagicAccessorImpl)ClassDefiner.defineClass(var13, var17, 0, var17.length, var1.getClassLoader()).newInstance();

                } catch (IllegalAccessException | InstantiationException var2) {

                    throw new InternalError(var2);

                }

            }

        });

    }

}

这段代码主要是动态生成SerializationConstructorAccessorImpl的class定义并返回实例(因为var7和var8都是true),ClassDefiner.defineClass方法的底层是调用本地C方法实现,类名是由MethodAccessorGenerator.generateName方法生成:

private static synchronized String generateName(boolean var0, boolean var1) {

    int var2;

    if (var0) {

        if (var1) {

            var2 = ++serializationConstructorSymnum;

            return "sun/reflect/GeneratedSerializationConstructorAccessor" + var2;

        } else {

            var2 = ++constructorSymnum;

            return "sun/reflect/GeneratedConstructorAccessor" + var2;

        }

    } else {

        var2 = ++methodSymnum;

        return "sun/reflect/GeneratedMethodAccessor" + var2;

    }

}

细心的同学会问为什么会每次调用都生成不同的类定义?因为MethodAccessorGenerator是需要多线程安全的,所以每次生成的类不能覆盖原其他线程生成的类定义,所以类名后面拼接序号保证不重复。至此,可以看出为什么压测会导致元空间内存增加的原因了。

本文来自:灰哥学堂

点赞收藏
技术小哥哥
请先登录,查看1条精彩评论吧
快去登录吧,你将获得
  • 浏览更多精彩评论
  • 和开发者讨论交流,共同进步

为你推荐

随机一门技术分享之Netty

随机一门技术分享之Netty

MappedByteBuffer VS FileChannel:从内核层面对比两者的性能差异

MappedByteBuffer VS FileChannel:从内核层面对比两者的性能差异

11
1