In our application almost all our datasources are dynamic datasources. If we load such a datasource using DataSource.forName when another datasource is executing we get some problems, it's a bit hard to be specific there is a lot going on.
This is the deadlock we are running into
So the first thread is loading a dynamic datasource, while the second thread is executing a DMI call that uses DataSource.fromXml("Some dynamic datasource" , (at this line: at com.genohm.slims.server.dmi.specific.RequestableDM I$1.execute(RequestableDMI.java:52))
So one of the locks comes from a synchronized method in PoolableDataSourceFactory
My question here is, to my untrained eye it looks like the synchronized here is not needed since both isSchemaLess and dsRequest are ThreadLocals. Removing this would solve my deadlock problem. Do you think it's correct to remove the synchronized here?
The SmartGWT version we are using is SmartClient Version: v9.0p_2013-12-06/PowerEdition Deployment (built 2013-12-06), but the same "synchronized" is still present in the latest builds.
This is the deadlock we are running into
Code:
Found one Java-level deadlock:
=============================
"btpool0-8":
waiting to lock monitor 7fc61f024848 (object 7989bc4c0, a com.isomorphic.datasource.PoolableDataSourceFactory),
which is held by "btpool0-5"
"btpool0-5":
waiting to lock monitor 7fc61f0248f0 (object 7e29882c8, a java.lang.Class),
which is held by "btpool0-8"
Java stack information for the threads listed above:
===================================================
"btpool0-8":
at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:129)
- waiting to lock <7989bc4c0> (a com.isomorphic.datasource.PoolableDataSourceFactory)
at com.isomorphic.datasource.PoolableDataSourceFactory.makeObject(PoolableDataSourceFactory.java:138)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1179)
at com.isomorphic.pool.PoolManager.borrowObject(PoolManager.java:84)
at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:95)
at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:287)
at com.isomorphic.datasource.BasicDataSource.findDataSource(BasicDataSource.java:2182)
at com.isomorphic.datasource.BasicDataSource.getType(BasicDataSource.java:2069)
at com.isomorphic.datasource.BasicDataSource.findElementType(BasicDataSource.java:1819)
at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:1195)
at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1107)
at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1091)
at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1052)
at com.isomorphic.datasource.DataSource.create(DataSource.java:1280)
at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:1619)
at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:1566)
at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:1399)
at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1107)
at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1066)
at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:1236)
at com.isomorphic.xml.XML.toDSRecords(XML.java:337)
at com.isomorphic.xml.XML.toDSRecords(XML.java:328)
at com.isomorphic.xml.XML.toDSRecords(XML.java:324)
at com.isomorphic.xml.XML.toDSRecords(XML.java:316)
at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:606)
at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:575)
at com.genohm.slims.server.datasource.generator.DataSourceConverter.convertToDs(DataSourceConverter.java:44)
at com.genohm.slims.server.datasource.generator.DataSourceConverter.getDataSource(DataSourceConverter.java:36)
at com.isomorphic.datasource.DataSource.getDynamicDataSource(DataSource.java:390)
- locked <7e29882c8> (a java.lang.Class for com.isomorphic.datasource.DataSource)
at com.isomorphic.datasource.DataSource.forName(DataSource.java:226)
at com.isomorphic.datasource.DataSource.forName(DataSource.java:220)
at com.isomorphic.datasource.DataSource.forName(DataSource.java:209)
at com.isomorphic.datasource.DataSource.forName(DataSource.java:205)
at com.genohm.slims.server.dmi.specific.RequestableDMI$1.execute(RequestableDMI.java:52)
at com.genohm.slims.server.dmi.specific.RequestableDMI$1.execute(RequestableDMI.java:1)
at com.genohm.slims.server.dmi.DSResponseUtil.wrapServiceCall(DSResponseUtil.java:144)
at com.genohm.slims.server.dmi.specific.RequestableDMI.getAvailableRequestables(RequestableDMI.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.isomorphic.base.Reflection.adaptArgsAndInvoke(Reflection.java:973)
at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:416)
at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:64)
at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:2427)
at com.isomorphic.servlet.IDACall.handleDSRequest(IDACall.java:215)
at com.isomorphic.servlet.IDACall.processRPCTransaction(IDACall.java:172)
at com.isomorphic.servlet.IDACall.processRequest(IDACall.java:137)
at com.isomorphic.servlet.IDACall.doPost(IDACall.java:73)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1097)
at com.isomorphic.servlet.CompressionFilter.doFilter(CompressionFilter.java:260)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:322)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:184)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:155)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.__handle(ContextHandler.java:729)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:843)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:647)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)
"btpool0-5":
at com.isomorphic.datasource.DataSource.getDynamicDataSource(DataSource.java:363)
- waiting to lock <7e29882c8> (a java.lang.Class for com.isomorphic.datasource.DataSource)
at com.isomorphic.datasource.DataSource.forName(DataSource.java:226)
at com.isomorphic.datasource.DataSource.forName(DataSource.java:220)
at com.isomorphic.datasource.DataSource.forName(DataSource.java:209)
at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:130)
- locked <7989bc4c0> (a com.isomorphic.datasource.PoolableDataSourceFactory)
at com.isomorphic.pool.PoolManager.borrowUnpooledObject(PoolManager.java:123)
at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:93)
at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:86)
at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:132)
at com.isomorphic.servlet.DataSourceLoader.doGet(DataSourceLoader.java:96)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1097)
at com.isomorphic.servlet.CompressionFilter.doFilter(CompressionFilter.java:247)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:322)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:184)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:155)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.__handle(ContextHandler.java:729)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:829)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)So one of the locks comes from a synchronized method in PoolableDataSourceFactory
Code:
public synchronized Object makeUnpooledObject(Object key)
throws Exception
{
Boolean isSchemaless = (Boolean)this.isSchemaless.get();
return DataSource.forName(key.toString(), (DSRequest)this.dsRequest.get(), isSchemaless == null ? false : isSchemaless.booleanValue());
}The SmartGWT version we are using is SmartClient Version: v9.0p_2013-12-06/PowerEdition Deployment (built 2013-12-06), but the same "synchronized" is still present in the latest builds.