Thursday, January 08, 2009

One thread lock issue sloved, another rises...

Recently during performing a load testing to our application, after taking stack trace from back-end, it appears 90% of http threads are locked as this kind of pattern:

"http-0.0.0.0-8080-12" daemon prio=6 tid=0x4bc5f620 nid=0xc54 waiting for monitor entry [0x4f4de000..0x4f4dfcec]
at java.lang.ref.ReferenceQueue.poll(ReferenceQueue.java:81)
- waiting to lock <0x07e4b968> (a java.lang.ref.ReferenceQueue$Lock)
at java.io.ObjectStreamClass.processQueue(ObjectStreamClass.java:2206)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:253)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1035)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
..........

Looking for help through the popular search engine, come across to this thread:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6525425
just exactly the same issue we met!!
we are using JDK1.5.0._07 version on our performance environment (BTW, we are not crazy to do version upgrade stuff :) )
this lock issue has been fixed in JDK1.5.0._14, look at the release notes:http://java.sun.com/j2se/1.5.0/ReleaseNotes.html#150_14

after validation, we find that the issue has really been fixed, there is no one thread lock as previous pattern, and throughput improves nearly 20% (under a relative high user load condition) ,HOWEVER, there is another type of lock rising almost 90% among all http threads.... Pattern 2:

"http-0.0.0.0-8080-48" daemon prio=6 tid=0x4dcc3808 nid=0xb0c waiting for monitor entry [0x5208c000..0x5208f9ec]
at org.jboss.metadata.WebMetaData.getRunAsIdentity(WebMetaData.java:511)
- waiting to lock <0x24e8c7e0> (a java.util.HashMap)
at org.jboss.web.tomcat.security.RunAsListener.instanceEvent(RunAsListener.java:67)
at org.apache.catalina.util.InstanceSupport.fireInstanceEvent(InstanceSupport.java:295)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:676)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:574)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:499)
at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:966)
at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:602)
at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:99)
at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:135)
........

we are targeting to look at the source code of third party, hope to solve it luckily, one of source code related:
http://www.java2s.com/Open-Source/Java-Document/EJB-Server-JBoss-4.2.1/tomcat/org/jboss/web/tomcat/security/RunAsListener.java.htm

No comments:

Post a Comment