性能文章>消失的死锁>

消失的死锁原创

2年前
1192700

问题描述

如果java层面发生了死锁,当我们使用jstack命令的时候其实是可以将死锁的信息给dump出来的,在dump结果的最后会有类似Found one Java-level deadlock:的关键字,接着会把发生死锁的线程的堆栈及对应的同步锁给打印出来,这次碰到一个系统就发生类似的问题,不过这个dump文档里虽然提到了如下的死锁信息:

Found one Java-level deadlock:
=============================
"worker-1-thread-121":
  waiting to lock monitor 0x00007f3758209dc8 (object 0x0000000764cd2b20, a java.util.concurrent.ConcurrentHashMap),
  which is held by "HSFBizProcessor-4-thread-4"
"HSFBizProcessor-4-thread-4":
  waiting to lock monitor 0x00007f3758289260 (object 0x000000076073ddc8, a com.rjb.test.extensions.equinox.KernelBundleClassLoader),
  which is held by "HSFBizProcessor-4-thread-5"
"HSFBizProcessor-4-thread-5":
  waiting to lock monitor 0x00007f3758253420 (object 0x00000007608e6fc8, a com.rjb.test.extensions.equinox.KernelBundleClassLoader),
  which is held by "HSFBizProcessor-4-thread-4"

但是我们在堆栈里搜索对应的锁的时候并没发现,也就是上面提到的

object 0x00000007608e6fc8 which is held by "HSFBizProcessor-4-thread-4"

我们在HSFBizProcessor-4-thread-4这个线程的堆栈里并没有看到对应的持锁信息。

附上线程dump详情

Found one Java-level deadlock:
=============================
"worker-1-thread-121":
  waiting to lock monitor 0x00007f3758209dc8 (object 0x0000000764cd2b20, a java.util.concurrent.ConcurrentHashMap),
  which is held by "HSFBizProcessor-4-thread-4"
"HSFBizProcessor-4-thread-4":
  waiting to lock monitor 0x00007f3758289260 (object 0x000000076073ddc8, a com.rjb.test.extensions.equinox.KernelBundleClassLoader),
  which is held by "HSFBizProcessor-4-thread-5"
"HSFBizProcessor-4-thread-5":
  waiting to lock monitor 0x00007f3758253420 (object 0x00000007608e6fc8, a com.rjb.test.extensions.equinox.KernelBundleClassLoader),
  which is held by "HSFBizProcessor-4-thread-4"

Java stack information for the threads listed above:
===================================================
"worker-1-thread-121":
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:180)
    - waiting to lock <0x0000000764cd2b20> (a java.util.concurrent.ConcurrentHashMap)
    at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:455)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:317)
    ......
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
"HSFBizProcessor-4-thread-4":
    at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLoadedClass(Unknown Source)
    - waiting to lock <0x000000076073ddc8> (a com.rjb.test.extensions.equinox.KernelBundleClassLoader)
    at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.SingleSourcePackage.loadClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(Unknown Source)
    at com.rjb.test.extensions.equinox.KernelBundleClassLoader.loadClass(KernelBundleClassLoader.java:121)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at org.springframework.scripting.groovy.GroovyScriptFactory.executeScript(GroovyScriptFactory.java:238)
    ......
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
"HSFBizProcessor-4-thread-5":
    at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLoadedClass(Unknown Source)
    - waiting to lock <0x00000007608e6fc8> (a com.rjb.test.extensions.equinox.KernelBundleClassLoader)
    at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.buddy.DependentPolicy.loadClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.buddy.PolicyHandler.doBuddyClassLoading(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(Unknown Source)
    at com.rjb.test.extensions.equinox.KernelBundleClassLoader.loadClass(KernelBundleClassLoader.java:121)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:169)
    at groovy.lang.MetaClassRegistry$MetaClassCreationHandle.createWithCustomLookup(MetaClassRegistry.java:127)
    at groovy.lang.MetaClassRegistry$MetaClassCreationHandle.create(MetaClassRegistry.java:122)
    ......
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Found 1 deadlock.

类加载的问题?

首先应该怀疑类加载的问题,因为我们看到导致死锁的对象是一个classloader对象:

waiting to lock monitor 0x00007f3758289260 (object 0x000000076073ddc8, a com.rjb.test.extensions.equinox.KernelBundleClassLoader)

然后我们再来分析下堆栈

HSFBizProcessor-4-thread-4

"HSFBizProcessor-4-thread-4":
    at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLoadedClass(Unknown Source)
    - waiting to lock <0x000000076073ddc8> (a com.rjb.test.extensions.equinox.KernelBundleClassLoader)
    at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.SingleSourcePackage.loadClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(Unknown Source)
    at com.rjb.test.extensions.equinox.KernelBundleClassLoader.loadClass(KernelBundleClassLoader.java:121)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at org.springframework.scripting.groovy.GroovyScriptFactory.executeScript(GroovyScriptFactory.java:238)
    at org.springframework.scripting.groovy.GroovyScriptFactory.getScriptedObject(GroovyScriptFactory.java:185)

我这里只把关键的线程栈贴出来,从栈顶知道正在等一把锁:

    - waiting to lock <0x000000076073ddc8> (a com.rjb.test.extensions.equinox.KernelBundleClassLoader)

这把锁的对象是一个ClassLoader对象,我们找到对应的代码,确实存在synchronized的操作:

private Class<?> findLoadedClass(String classname) {
    if ((LOCK_CLASSNAME) || (this.isParallelClassLoader)) {
      boolean initialLock = lockClassName(classname);
      try {
        return this.classloader.publicFindLoaded(classname);
      } finally {
        if (initialLock)
          unlockClassName(classname);
      }
    }
    synchronized (this.classloader) {
      return this.classloader.publicFindLoaded(classname);
    }
  }

另外我们还知道它正在执行loadClass的动作,并且是从groovy调用来的,同样找到对应的代码:

protected Object executeScript(ScriptSource scriptSource, Class scriptClass)
    throws ScriptCompilationException
  {
    try
    {
      GroovyObject goo = (GroovyObject)scriptClass.newInstance();//line 238

      if (this.groovyObjectCustomizer != null)
      {
        this.groovyObjectCustomizer.customize(goo);
      }

      if ((goo instanceof Script))
      {
        return ((Script)goo).run();
      }

      return goo;
    }
    catch (InstantiationException ex)
    {
      throw new ScriptCompilationException(
        scriptSource, "Could not instantiate Groovy script class: " + scriptClass.getName(), ex);
    }
    catch (IllegalAccessException ex) {
      throw new ScriptCompilationException(
        scriptSource, "Could not access Groovy script constructor: " + scriptClass.getName(), ex);
    }
  }

执行到第238行的时候

GroovyObject goo = (GroovyObject)scriptClass.newInstance();//line 238

突然发现调用了

java.lang.ClassLoader.loadClass(ClassLoader.java:247)

而我们看到上面第238行的逻辑其实就是实例化一个对象,然后进行强转,我们看看对应的字节码:

 0: aload_2
 1: invokevirtual #164                // Method java/lang/Class.newInstance:()Ljava/lang/Object;
 4: checkcast     #168                // class groovy/lang/GroovyObject
 7: astore_3

其实就对应这么几条字节码指令,其实在jvm里当我们执行checkcast指令的时候会触发类加载的动作:

void TemplateTable::checkcast() {
    ...
    call_VM(rax, CAST_FROM_FN_PTR(address, InterpreterRuntime::quicken_io_cc));
    ...
}

IRT_ENTRY(void, InterpreterRuntime::quicken_io_cc(JavaThread* thread))
  // Force resolving; quicken the bytecode
  int which = get_index_u2(thread, Bytecodes::_checkcast);
  constantPoolOop cpool = method(thread)->constants();
  // We'd expect to assert that we're only here to quicken bytecodes, but in a multithreaded
  // program we might have seen an unquick'd bytecode in the interpreter but have another
  // thread quicken the bytecode before we get here.
  // assert( cpool->tag_at(which).is_unresolved_klass(), "should only come here to quicken bytecodes" );
  klassOop klass = cpool->klass_at(which, CHECK);
  thread->set_vm_result(klass);
IRT_END

klassOop klass_at(int which, TRAPS) {
    constantPoolHandle h_this(THREAD, this);
    return klass_at_impl(h_this, which, CHECK_NULL);
}

klassOop constantPoolOopDesc::klass_at_impl(constantPoolHandle this_oop, int which, TRAPS) {
    ...
    klassOop k_oop = SystemDictionary::resolve_or_fail(name, loader, h_prot, true, THREAD);
    ...
}    

//SystemDictionary::resolve_or_fail最终会调用到下面这个方法
klassOop SystemDictionary::resolve_instance_class_or_null(Symbol* name, Handle class_loader, Handle protection_domain, TRAPS) {
  ...
  // Class is not in SystemDictionary so we have to do loading.
  // Make sure we are synchronized on the class loader before we proceed
  Handle lockObject = compute_loader_lock_object(class_loader, THREAD);
  check_loader_lock_contention(lockObject, THREAD);
  ObjectLocker ol(lockObject, THREAD, DoObjectLock);
  ...
  //此时会调用ClassLoader.loadClass来加载类了
  ...
}

Handle SystemDictionary::compute_loader_lock_object(Handle class_loader, TRAPS) {
  // If class_loader is NULL we synchronize on _system_loader_lock_obj
  if (class_loader.is_null()) {
    return Handle(THREAD, _system_loader_lock_obj);
  } else {
    return class_loader;
  }
}

SystemDictionary::resolve_instance_class_or_null这个方法非常关键了,在里面我们看到会获取一把锁ObjectLocker,其相当于我们java代码里的synchronized关键字,而对象对应的是lockObject,这个对象是上面的SystemDictionary::compute_loader_lock_object方法返回的,从代码可知只要不是bootstrapClassloader加载的类就会返回当前classloader对象,也就是说当我们在加载一个类的时候其实是会持有当前类加载对象的锁的,在获取了这把锁之后就会调用ClassLoader.loadClass来加载类了。这其实就解释了HSFBizProcessor-4-thread-4这个线程为什么持有了

object 0x00000007608e6fc8, a com.rjb.test.extensions.equinox.KernelBundleClassLoader

这个类加载的锁,不过遗憾的是因为这把锁不是java层面来显示加载的,因此我们在jstack线程dump的输出里居然看不到这把锁的存在.

HSFBizProcessor-4-thread-5

先上堆栈:

"HSFBizProcessor-4-thread-5":
    at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLoadedClass(Unknown Source)
    - waiting to lock <0x00000007608e6fc8> (a com.rjb.test.extensions.equinox.KernelBundleClassLoader)
    at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.buddy.DependentPolicy.loadClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.buddy.PolicyHandler.doBuddyClassLoading(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(Unknown Source)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(Unknown Source)
    at com.rjb.test.extensions.equinox.KernelBundleClassLoader.loadClass(KernelBundleClassLoader.java:121)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:169)

这个线程栈其实和之前那个线程差不多,只是等的锁不一样,另外触发类加载的动作是Class.forName,获取大家也猜到了,其实是在下面两行堆栈之间同样获取了一把类加载器的锁

    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)

这里的代码我也不细贴了,最终调用的jvm里的方法都是一样的,获取锁的逻辑也是一样的

总结

想象下这种场景,两个线程分别使用不同的classloader对两个类进行类加载,然而由于osgi类加载机制的缘故,在loadClass过程中可能会委托给别的classloader去加载,而正巧,这两个线程在获取当前classloader的锁之后,然后分别委托对方的classloader去加载,可以看到文章开头列的那个findLoadedClass方法,而synchronized的那个classloader正好是对方的classloader,从而导致了死锁

分类:
标签:
请先登录,再评论

暂无回复,快来写下第一个回复吧~

为你推荐

类初始化导致死锁
一张图简单描述死锁 如上图,Thread1 拿到了 object1,Thread2 拿到了 object2,但是现在 Thread1 需要拿到 object2 的锁才能继续往下,Thread2 又要拿到 object1 才能继续往下
字符串字面量长度是有限制的
前言 偶然在一次单元测试中写了一个非常长的字符串字面量。 正文 在一次单元测试中,我写了一个很长的字符串字面量,大概10万个字符左右,编译时,编译器给出了异常告警 `java: constant
多次字符串相加一定要用StringBuilder而不用-吗?
今天在写一个读取Java class File并进行分析的Demo时,偶然发现了下面这个场景(基于oracle jdk 1.8.0_144): ``` package test; public c
如何通过反射获得方法的真实参数名(以及扩展研究)
前段时间,在做一个小的工程时,遇到了需要通过反射获得方法真实参数名的场景,在这里我遇到了一些小小的问题,后来在部门老大的指导下,我解决了这个问题。通过解决这个问题,附带着我了解到了很多新的知识,我觉得
高吞吐、低延迟 Java 应用的 GC 优化实践
本篇原文作者是 LinkedIn 的 Swapnil Ghike,这篇文章讲述了 LinkedIn 的 Feed 产品的 GC 优化过程,虽然文章写作于 April 8, 2014,但其中的很多内容和
「每日五分钟,玩转 JVM」:久识你名,初居我心
聊聊 JVMJVM,一个熟悉又陌生的名词,从认识Java的第一天起,我们就会听到这个名字,在参加工作的前一两年,面试的时候还会经常被问到JDK,JRE,JVM这三者的区别。JVM可以说和我们是老朋友了
据说99.99%的人都会答错的类加载的问题
概述首先还是把问题抛给大家,这个问题也是我厂同学在做一个性能分析产品的时候碰到的一个问题。 同一个类加载器对象是否可以加载同一个类文件多次并且得到多个Class对象而都可以被java层使用吗请仔细注意
Java多线程——并发测试
编写并发程序时候,可以采取和串行程序相同的编程方式。唯一的难点在于,并发程序存在不确定性,这种不确定性会令程序出错的地方远比串行程序多,出现的方式也没有固定规则。那么如何在测试中,尽可能的暴露出这些问