一次full gc的排查经历原创
4年前
9155111
问题描述
1月22日准备发布设备指纹新版本迭代过程中，在预发环境进行采集请求的压测发现了个现象，监控中的RT存在规律的尖点。
接着又尝试master代码，发现不存在这个问题，那基本确认是新代码引入的问题了。
问题定位
猜测是full gc导致RT变长，用jstat命令查看果然不出所料。
那么为什么会出现频繁的full gc呢？接下来开始漫长的排查过程：
代码定位
由于新版本改动点比较多，所以我们尝试一步步还原代码压测，最终发现是kryo包从2.16升级到2.24.0导致了这个现象。
堆内存排查
full gc，第一反应都会以为是由于堆中老年代对象过多引起的，所以二话不说开始执行jmap dump堆内存，结果4G多的内存下载下来分析后并无头绪。
jfr分析排查
执行jmc命令记录jfr文件进行分析，结果发现在full gc前后metaspace明显减少了（项目使用的是java1.8）：
并且在类卸载那一栏中有非常多的sun.reflect.GeneratedSerializationConstructorAccessor被卸载：
那么应该是元空间中频繁加载了过多动态类导致的full gc。再次用jstat命令进行确认了一遍，发现在每一次full gc后metaspace都有明显的变少，问题基本确认。
问题深入
那么kryo包的这2个版本有什么区别呢？为什么会频繁使用动态类加载？又开始了漫长的分析过程：
kryo是什么，怎么用
Kryo 是一个快速序列化/反序列化工具，其使用了字节码生成机制（底层依赖了ASM库），因此具有比较好的运行速度。
项目中使用kryo序列化代码：
public byte[] kryoSerialize(AerospikeItemNewSession aerospikeItemNewSession) {
byte saveResultBytes[] = null;
ByteArrayOutputStream byt1 = new ByteArrayOutputStream();
Output output = new Output(byt1);
try {
Kryo kryo = new Kryo();
kryo.setReferences(false);//是否需要支持循环引用
kryo.setRegistrationRequired(false);//是否开启注册行为
kryo.setInstantiatorStrategy(new StdInstantiatorStrategy());//反序列化策略
kryo.register(AerospikeItemNewSession.class);
kryo.writeObject(output, aerospikeItemNewSession);
output.flush();
saveResultBytes = byt1.toByteArray();
} catch (Throwable ex) {
logger.error("kryoSerialize occur,");
} finally {
try {
byt1.flush();
byt1.close();
output.close();
} catch (IOException e) {
logger.error("kryoSerialize close object occur,", e);
}
}
return saveResultBytes;
}
项目中使用kryo反序列化代码：
public String kryoDeserialize(byte[] saveResultBytes) {
String result = "";
AerospikeItemNewSession aerospikeItemNewSession;
ByteArrayInputStream bis1 = new ByteArrayInputStream(saveResultBytes);
Input input = new Input(bis1);
try {
Kryo kryo = new Kryo();
kryo.setReferences(false);//是否需要支持循环引用
kryo.setRegistrationRequired(false);//是否开启注册行为
kryo.setInstantiatorStrategy(new StdInstantiatorStrategy());//反序列化策略
kryo.register(AerospikeItemNewSession.class);
if ((aerospikeItemNewSession = kryo.readObject(input, AerospikeItemNewSession.class)) != null) {
if (aerospikeItemNewSession.isCompressed()) {
result = CompressUtil.ungzip(aerospikeItemNewSession.getValue());
} else {
result = new String(aerospikeItemNewSession.getValue(), "utf-8");
}
}
} catch (Exception ex) {
logger.error("kryoDeserialize occur,", ex);
} finally {
try {
bis1.close();
input.close();
} catch (IOException e) {
logger.error("kryoDeserialize close object occur,", e);
}
}
return result;
}
进入误区
因为是设备指纹的采集请求，所以想当然认为是kryo.writeObject(output, aerospikeItemNewSession);序列化代码出现问题。
接着开始进行源码研究和调试，发现并没有出现动态类创建的方法和代码，后面又进行了如下尝试：
-
启动时用class.forName将AerospikeItemNewSession类装载一次, 无用
-
将AerospikeItemNewSession类改成静态类，无用
-
将AerospikeItemNewSession外部public类（原先是写在同级类下，只有同个包访问权限），无用
-
将类使用到的序列化器FieldSerializer改成2.16版本的代码，无用
地毯式搜索
既然是反射动态生成类引起的问题，所以干脆在kryo代码中全局搜索reflect相关的代码，发现kyro.java下的一段代码有点可疑：
public ObjectInstantiator newInstantiatorOf (final Class type) {
if (!Util.isAndroid) {
// Use ReflectASM if the class is not a non-static member class.
Class enclosingType = type.getEnclosingClass();
boolean isNonStaticMemberClass = enclosingType != null && type.isMemberClass()
&& !Modifier.isStatic(type.getModifiers());
if (!isNonStaticMemberClass) {
try {
final ConstructorAccess access = ConstructorAccess.get(type);
return new ObjectInstantiator() {
public Object newInstance () {
try {
return access.newInstance();
} catch (Exception ex) {
throw new KryoException("Error constructing instance of class: " + className(type), ex);
}
}
};
} catch (Exception ignored) {
}
}
}
// Reflection.
try {
Constructor ctor;
try {
ctor = type.getConstructor((Class[])null);
} catch (Exception ex) {
ctor = type.getDeclaredConstructor((Class[])null);
ctor.setAccessible(true);
}
final Constructor constructor = ctor;
return new ObjectInstantiator() {
public Object newInstance () {
try {
return constructor.newInstance();
} catch (Exception ex) {
throw new KryoException("Error constructing instance of class: " + className(type), ex);
}
}
};
} catch (Exception ignored) {
}
if (fallbackStrategy == null) {
if (type.isMemberClass() && !Modifier.isStatic(type.getModifiers()))
throw new KryoException("Class cannot be created (non-static member class): " + className(type));
else
throw new KryoException("Class cannot be created (missing no-arg constructor): " + className(type));
}
// InstantiatorStrategy.
return fallbackStrategy.newInstantiatorOf(type);
}
接着往前寻找他的调用栈，发现这是在反序列化的时候使用的，用于生成AerospikeItemNewSession实例，突然意识到是不是反序列也需要反射生成实例呢，而采集请求也调用到了这块，马上去对比了下2.16的代码，发现protected ObjectInstantiator newInstantiator (final Class type)方法有差别：
2.16的代码是这样的:
/** Returns a new instantiator for creating new instances of the specified type. By default, an instantiator is returned that
* uses reflection if the class has a zero argument constructor, an exception is thrown. If a
* {@link #setInstantiatorStrategy(InstantiatorStrategy) strategy} is set, it will be used instead of throwing an exception. */
protected ObjectInstantiator newInstantiator (final Class type) {
if (!Util.isAndroid) {
// ReflectASM.
try {
final ConstructorAccess access = ConstructorAccess.get(type);
return new ObjectInstantiator() {
public Object newInstance () {
try {
return access.newInstance();
} catch (Exception ex) {
throw new KryoException("Error constructing instance of class: " + className(type), ex);
}
}
};
} catch (Exception ignored) {
}
}
// Reflection.
try {
Constructor ctor;
try {
ctor = type.getConstructor((Class[])null);
} catch (Exception ex) {
ctor = type.getDeclaredConstructor((Class[])null);
ctor.setAccessible(true);
}
final Constructor constructor = ctor;
return new ObjectInstantiator() {
public Object newInstance () {
try {
return constructor.newInstance();
} catch (Exception ex) {
throw new KryoException("Error constructing instance of class: " + className(type), ex);
}
}
};
} catch (Exception ignored) {
}
if (strategy == null) {
if (type.isMemberClass() && !Modifier.isStatic(type.getModifiers()))
throw new KryoException("Class cannot be created (non-static member class): " + className(type));
else
throw new KryoException("Class cannot be created (missing no-arg constructor): " + className(type));
}
// InstantiatorStrategy.
return strategy.newInstantiatorOf(type);
}
2.24.0的代码是这样的:
/** Returns a new instantiator for creating new instances of the specified type. By default, an instantiator is returned that
* uses reflection if the class has a zero argument constructor, an exception is thrown. If a
* {@link #setInstantiatorStrategy(InstantiatorStrategy) strategy} is set, it will be used instead of throwing an exception. */
protected ObjectInstantiator newInstantiator (final Class type) {
// InstantiatorStrategy.
return strategy.newInstantiatorOf(type);
}
2.24.0里把2.16里return strategy.newInstantiatorOf(type);上面的很大一段代码挪到了DefaultInstantiatorStrategy的实现里了，而项目中显式设置了StdInstantiatorStrategy策略，因此2.24.0就不会执行那段代码而直接采用StdInstantiatorStrategy创建实例化器，2.16会执行上面那段代码创建实例化器。为了证实我的猜想，分别调试了2个版本的代码，发现确实如此。
2.16版本反序列调用链路：
Kryo.readObject (Input input, Class<T> type) -> FieldSerializer.read (Kryo kryo, Input input, Class<T> type) -> FieldSerializer.create (Kryo kryo, Input input, Class<T> type) -> Kryo.newInstance (Class<T> type) -> Kryo.newInstantiator (final Class type)
2.24版本反序列调用链路：
Kryo.readObject (Input input, Class<T> type) -> FieldSerializer.read (Kryo kryo, Input input, Class<T> type) -> FieldSerializer.create (Kryo kryo, Input input, Class<T> type) -> Kryo.newInstance (Class<T> type) -> Kryo.newInstantiator (final Class type) -> StdInstantiatorStrategy.newInstantiatorOf(Class type)
问题解决
基于上面的分析思路，去掉策略设置kryo.setInstantiatorStrategy(new StdInstantiatorStrategy());，默认就会使用DefaultInstantiatorStrategy执行，执行的逻辑和2.16保持一致，再进行一次压测之后不再出现full gc。
kryo源码分析
为什么执行了StdInstantiatorStrategy会造成频繁创建class？
静态class定义创建和加载
从上面的分析可以看出2.16中Kryo.newInstantiator (final Class type)方法和2.24中DefaultInstantiatorStrategy类会优先执行一段如下的代码逻辑：
if (!Util.isAndroid) {
// Use ReflectASM if the class is not a non-static member class.
Class enclosingType = type.getEnclosingClass();
boolean isNonStaticMemberClass = enclosingType != null && type.isMemberClass()
&& !Modifier.isStatic(type.getModifiers());
if (!isNonStaticMemberClass) {
try {
final ConstructorAccess access = ConstructorAccess.get(type);
return new ObjectInstantiator() {
public Object newInstance () {
try {
return access.newInstance();
} catch (Exception ex) {
throw new KryoException("Error constructing instance of class: " + className(type), ex);
}
}
};
} catch (Exception ignored) {
}
}
}
这段代码返回了ObjectInstantiator类型，里面newInstatnce方法就是用于创建反序列化类。代码里ConstructorAccess类是一个反序列化实例的构造器访问类，可以理解为一个能够构造实例的工厂类（和ObjectInstantiator有点像），那么这个方法final ConstructorAccess access = ConstructorAccess.get(type);是怎么获取和构造这个工厂呢？进入get方法查看源码：
public static <T> ConstructorAccess<T> get(Class<T> type) {
Class enclosingType = type.getEnclosingClass();
boolean isNonStaticMemberClass = enclosingType != null && type.isMemberClass() && !Modifier.isStatic(type.getModifiers());
String className = type.getName();
String accessClassName = className + "ConstructorAccess";
if (accessClassName.startsWith("java.")) {
accessClassName = "reflectasm." + accessClassName;
}
Class accessClass = null;
AccessClassLoader loader = AccessClassLoader.get(type);
synchronized(loader) {
try {
accessClass = loader.loadClass(accessClassName);
} catch (ClassNotFoundException var18) {
String accessClassNameInternal = accessClassName.replace('.', '/');
String classNameInternal = className.replace('.', '/');
boolean isPrivate = false;
String enclosingClassNameInternal;
Constructor constructor;
if (!isNonStaticMemberClass) {
enclosingClassNameInternal = null;
try {
constructor = type.getDeclaredConstructor((Class[])null);
isPrivate = Modifier.isPrivate(constructor.getModifiers());
} catch (Exception var17) {
throw new RuntimeException("Class cannot be created (missing no-arg constructor): " + type.getName(), var17);
}
if (isPrivate) {
throw new RuntimeException("Class cannot be created (the no-arg constructor is private): " + type.getName());
}
} else {
enclosingClassNameInternal = enclosingType.getName().replace('.', '/');
try {
constructor = type.getDeclaredConstructor(enclosingType);
isPrivate = Modifier.isPrivate(constructor.getModifiers());
} catch (Exception var16) {
throw new RuntimeException("Non-static member class cannot be created (missing enclosing class constructor): " + type.getName(), var16);
}
if (isPrivate) {
throw new RuntimeException("Non-static member class cannot be created (the enclosing class constructor is private): " + type.getName());
}
}
ClassWriter cw = new ClassWriter(0);
cw.visit(196653, 33, accessClassNameInternal, (String)null, "com/esotericsoftware/reflectasm/ConstructorAccess", (String[])null);
insertConstructor(cw);
insertNewInstance(cw, classNameInternal);
insertNewInstanceInner(cw, classNameInternal, enclosingClassNameInternal);
cw.visitEnd();
accessClass = loader.defineClass(accessClassName, cw.toByteArray());
}
}
try {
ConstructorAccess<T> access = (ConstructorAccess)accessClass.newInstance();
access.isNonStaticMemberClass = isNonStaticMemberClass;
return access;
} catch (Exception var15) {
throw new RuntimeException("Error constructing constructor access class: " + accessClassName, var15);
}
}
可以看到这段代码主要采用的是classloader方式defineClass和loadClass，accessClassName是固定的（className+“ConstructorAccess”），第一次调用accessClass = loader.loadClass(accessClassName);的地方会抛出ClassNotFoundException异常，异常捕获后开始创建accessClass，只要被序列化的类不是私有的内部类，则可以通过这种方式创建出工厂类accessClass，然后accessClass = loader.defineClass(accessClassName, cw.toByteArray());定义到元空间中，以后loader.loadClass(accessClassName)的调用则加载同一个类定义。
动态类创建
2.16中Kryo.newInstantiator (final Class type)方法和2.24中DefaultInstantiatorStrategy类的最后会调用strategy去动态生成class定义，也就是说只有当静态class创建加载失败的时候，才会执行策略，接下来我们看下StdInstantiatorStrategy.java的实现（以下是2.24.0的代码，2.16也是类似）：
public <T> ObjectInstantiator<T> newInstantiatorOf(Class<T> type) {
if(PlatformDescription.isThisJVM(SUN) || PlatformDescription.isThisJVM(OPENJDK)) {
// The UnsafeFactoryInstantiator would also work. But according to benchmarks, it is 2.5
// times slower. So I prefer to use this one
return new SunReflectionFactoryInstantiator<T>(type);
}
else if(PlatformDescription.isThisJVM(JROCKIT)) {
if(VM_VERSION.startsWith("1.4")) {
// JRockit vendor version will be RXX where XX is the version
// Versions prior to 26 need special handling
// From R26 on, java.vm.version starts with R
if(!VENDOR_VERSION.startsWith("R")) {
// On R25.1 and R25.2, ReflectionFactory should work. Otherwise, we must use the
// Legacy instantiator.
if(VM_INFO == null || !VM_INFO.startsWith("R25.1") || !VM_INFO.startsWith("R25.2")) {
return new JRockitLegacyInstantiator<T>(type);
}
}
}
// After that, JRockit became compliant with HotSpot
return new SunReflectionFactoryInstantiator<T>(type);
}
else if(PlatformDescription.isThisJVM(DALVIK)) {
if(ANDROID_VERSION <= 10) {
// Android 2.3 Gingerbread and lower
return new Android10Instantiator<T>(type);
}
if(ANDROID_VERSION <= 17) {
// Android 3.0 Honeycomb to 4.2 Jelly Bean
return new Android17Instantiator<T>(type);
}
// Android 4.3 and higher (hopefully)
return new Android18Instantiator<T>(type);
}
else if(PlatformDescription.isThisJVM(GNU)) {
return new GCJInstantiator<T>(type);
}
else if(PlatformDescription.isThisJVM(PERC)) {
return new PercInstantiator<T>(type);
}
// Fallback instantiator, should work with most modern JVM
return new UnsafeFactoryInstantiator<T>(type);
}
可以看出如果是sun的jdk会执行return new SunReflectionFactoryInstantiator<T>(type);，SunReflectionFactoryInstantiator就是ObjectInstantiator的实现类，内部维护了一个private final Constructor mungedConstructor;属性，这是一个构造函数类实例，可以理解为和5.1中的ConstructorAccess工厂类类似，能够实例化反序列化类。mungedConstructor属性最终是被ReflectionFactory.generateConstructor生成:
private final Constructor<?> generateConstructor(Class<?> var1, Constructor<?> var2) {
SerializationConstructorAccessorImpl var3 = (new MethodAccessorGenerator()).generateSerializationConstructor(var1, var2.getParameterTypes(), var2.getExceptionTypes(), var2.getModifiers(), var2.getDeclaringClass());
Constructor var4 = this.newConstructor(var2.getDeclaringClass(), var2.getParameterTypes(), var2.getExceptionTypes(), var2.getModifiers(), langReflectAccess().getConstructorSlot(var2), langReflectAccess().getConstructorSignature(var2), langReflectAccess().getConstructorAnnotations(var2), langReflectAccess().getConstructorParameterAnnotations(var2));
this.setConstructorAccessor(var4, var3);
var4.setAccessible(true);
return var4;
}
而mungedConstructor中的constructorAccessor（Constructor中的newInstance方法都会调用constructorAccessor.newInstatnce）是由MethodAccessorGenerator.generate方法生成，代码如下：
private MagicAccessorImpl generate(final Class<?> var1, String var2, Class<?>[] var3, Class<?> var4, Class<?>[] var5, int var6, boolean var7, boolean var8, Class<?> var9) {
ByteVector var10 = ByteVectorFactory.create();
this.asm = new ClassFileAssembler(var10);
this.declaringClass = var1;
this.parameterTypes = var3;
this.returnType = var4;
this.modifiers = var6;
this.isConstructor = var7;
this.forSerialization = var8;
this.asm.emitMagicAndVersion();
short var11 = 42;
boolean var12 = this.usesPrimitiveTypes();
if (var12) {
var11 = (short)(var11 + 72);
}
if (var8) {
var11 = (short)(var11 + 2);
}
var11 += (short)(2 * this.numNonPrimitiveParameterTypes());
this.asm.emitShort(add(var11, (short)1));
final String var13 = generateName(var7, var8);
this.asm.emitConstantPoolUTF8(var13);
this.asm.emitConstantPoolClass(this.asm.cpi());
this.thisClass = this.asm.cpi();
if (var7) {
if (var8) {
this.asm.emitConstantPoolUTF8("sun/reflect/SerializationConstructorAccessorImpl");
} else {
this.asm.emitConstantPoolUTF8("sun/reflect/ConstructorAccessorImpl");
}
} else {
this.asm.emitConstantPoolUTF8("sun/reflect/MethodAccessorImpl");
}
this.asm.emitConstantPoolClass(this.asm.cpi());
this.superClass = this.asm.cpi();
this.asm.emitConstantPoolUTF8(getClassName(var1, false));
this.asm.emitConstantPoolClass(this.asm.cpi());
this.targetClass = this.asm.cpi();
short var14 = 0;
if (var8) {
this.asm.emitConstantPoolUTF8(getClassName(var9, false));
this.asm.emitConstantPoolClass(this.asm.cpi());
var14 = this.asm.cpi();
}
this.asm.emitConstantPoolUTF8(var2);
this.asm.emitConstantPoolUTF8(this.buildInternalSignature());
this.asm.emitConstantPoolNameAndType(sub(this.asm.cpi(), (short)1), this.asm.cpi());
if (this.isInterface()) {
this.asm.emitConstantPoolInterfaceMethodref(this.targetClass, this.asm.cpi());
} else if (var8) {
this.asm.emitConstantPoolMethodref(var14, this.asm.cpi());
} else {
this.asm.emitConstantPoolMethodref(this.targetClass, this.asm.cpi());
}
this.targetMethodRef = this.asm.cpi();
if (var7) {
this.asm.emitConstantPoolUTF8("newInstance");
} else {
this.asm.emitConstantPoolUTF8("invoke");
}
this.invokeIdx = this.asm.cpi();
if (var7) {
this.asm.emitConstantPoolUTF8("([Ljava/lang/Object;)Ljava/lang/Object;");
} else {
this.asm.emitConstantPoolUTF8("(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;");
}
this.invokeDescriptorIdx = this.asm.cpi();
this.nonPrimitiveParametersBaseIdx = add(this.asm.cpi(), (short)2);
for(int var15 = 0; var15 < var3.length; ++var15) {
Class var16 = var3[var15];
if (!isPrimitive(var16)) {
this.asm.emitConstantPoolUTF8(getClassName(var16, false));
this.asm.emitConstantPoolClass(this.asm.cpi());
}
}
this.emitCommonConstantPoolEntries();
if (var12) {
this.emitBoxingContantPoolEntries();
}
if (this.asm.cpi() != var11) {
throw new InternalError("Adjust this code (cpi = " + this.asm.cpi() + ", numCPEntries = " + var11 + ")");
} else {
this.asm.emitShort((short)1);
this.asm.emitShort(this.thisClass);
this.asm.emitShort(this.superClass);
this.asm.emitShort((short)0);
this.asm.emitShort((short)0);
this.asm.emitShort((short)2);
this.emitConstructor();
this.emitInvoke();
this.asm.emitShort((short)0);
var10.trim();
final byte[] var17 = var10.getData();
return (MagicAccessorImpl)AccessController.doPrivileged(new PrivilegedAction<MagicAccessorImpl>() {
public MagicAccessorImpl run() {
try {
return (MagicAccessorImpl)ClassDefiner.defineClass(var13, var17, 0, var17.length, var1.getClassLoader()).newInstance();
} catch (IllegalAccessException | InstantiationException var2) {
throw new InternalError(var2);
}
}
});
}
}
这段代码主要是动态生成SerializationConstructorAccessorImpl的class定义并返回实例（因为var7和var8都是true），ClassDefiner.defineClass方法的底层是调用本地C方法实现，类名是由MethodAccessorGenerator.generateName方法生成：
private static synchronized String generateName(boolean var0, boolean var1) {
int var2;
if (var0) {
if (var1) {
var2 = ++serializationConstructorSymnum;
return "sun/reflect/GeneratedSerializationConstructorAccessor" + var2;
} else {
var2 = ++constructorSymnum;
return "sun/reflect/GeneratedConstructorAccessor" + var2;
}
} else {
var2 = ++methodSymnum;
return "sun/reflect/GeneratedMethodAccessor" + var2;
}
}
细心的同学会问为什么会每次调用都生成不同的类定义？因为MethodAccessorGenerator是需要多线程安全的，所以每次生成的类不能覆盖原其他线程生成的类定义，所以类名后面拼接序号保证不重复。至此，可以看出为什么压测会导致元空间内存增加的原因了。
本文来自：灰哥学堂
点赞收藏
分类：
