Quantcast
Channel: SmartClient Forums
Viewing all articles
Browse latest Browse all 4756

Potential Thread Deadlock Within DataSourceManager.getDataSource()

$
0
0
SmartGWTPower v10.0p_2015-06-23/PowerEdition Development Only

We have hit a problem that we have not been able to produce a test case for since we don't know precisely what causes the issue. However, we have done some investigation and arrived at a point that we think you may help with.

In short, the problem is that the server side of our application stops responding. It tends to do this when relatively few (but usually more than one) people are logged in, and we believe we have traced it to a deadlock between two synchronized methods within the DataSourceManager. This appears only to have been a problem since we updated to recent versions of SmartGWT from v3.1 to v5.0.

To arrive at this conclusion we used VisualVM to monitor memory and thread utilization. When the server stopped responding we saw a deadlock condition which, summarized, looks like this:

Code:

"http-bio-8088-exec-66" - Thread t@357
  java.lang.Thread.State: BLOCKED
        at com.isomorphic.datasource.SimpleType.validateValue(SimpleType.java:89)
        - waiting to lock <42d5dfc4> (a com.isomorphic.datasource.SimpleType) owned by "http-bio-8088-exec-69" t@368
        at com.isomorphic.datasource.SimpleType.create(SimpleType.java:73)
        at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:1962)
        at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:1877)
        at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:1554)
        at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1225)
        at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1184)
        at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:1601)
        at com.isomorphic.xml.XML.toDSRecords(XML.java:363)
        at com.isomorphic.xml.XML.toDSRecords(XML.java:349)
        at com.isomorphic.xml.XML.toDSRecords(XML.java:345)
        at com.isomorphic.xml.XML.toDSRecords(XML.java:328)
        at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:966)
        at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:950)
        at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:919)
        at com.islandpacific.gui.server.customDataSource.IpDynamicDataSource.getIPDataSource(IpDynamicDataSource.java:121)
        at com.islandpacific.gui.server.customDataSource.IpDynamicDataSource.getDataSource(IpDynamicDataSource.java:106)
        at com.islandpacific.gui.server.customDataSource.IslandPacificDSLoader.processRequest(IslandPacificDSLoader.java:92)
        at com.isomorphic.servlet.DataSourceLoader.doPost(DataSourceLoader.java:101)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
        at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:156)


"http-bio-8088-exec-68" - Thread t@367
  java.lang.Thread.State: BLOCKED
        at com.isomorphic.datasource.SimpleType.validateValue(SimpleType.java:89)
        - waiting to lock <42d5dfc4> (a com.isomorphic.datasource.SimpleType) owned by "http-bio-8088-exec-69" t@368
        at com.isomorphic.datasource.SimpleType.create(SimpleType.java:73)
        at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:1962)
        at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:1877)
        at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:1554)
        at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1225)
        at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1184)
        at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:1601)
        at com.isomorphic.xml.XML.toDSRecords(XML.java:363)
        at com.isomorphic.xml.XML.toDSRecords(XML.java:349)
        at com.isomorphic.xml.XML.toDSRecords(XML.java:345)
        at com.isomorphic.xml.XML.toDSRecords(XML.java:328)
        at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:966)
        at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:950)
        at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:919)
        at com.islandpacific.gui.server.customDataSource.IpDynamicDataSource.getIPDataSource(IpDynamicDataSource.java:121)
        at com.islandpacific.gui.server.customDataSource.IpDynamicDataSource.getDataSource(IpDynamicDataSource.java:106)
        at com.isomorphic.datasource.DataSource.getDataSourceFromStack(DataSource.java:707)
        at com.isomorphic.datasource.DataSource.getDynamicDataSource(DataSource.java:680)
        - locked <3866e65f> (a java.lang.Class)
        at com.isomorphic.datasource.DataSource.forName(DataSource.java:342)
        at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:132)
        - locked <384d3cc3> (a com.isomorphic.datasource.PoolableDataSourceFactory)
        at com.isomorphic.pool.PoolManager.borrowUnpooledObject(PoolManager.java:129)
        at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:133)
        - locked <4abd797a> (a java.lang.Class)
        at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:86)
        at com.isomorphic.datasource.DSRequest.getDataSource(DSRequest.java:2216)
        at com.isomorphic.datasource.DSRequest.buildFieldData(DSRequest.java:3046)
        at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:2392)
        at com.islandpacific.gui.server.job.JobUtilities.nextJobNumber(JobUtilities.java:97)
        at com.islandpacific.gui.server.job.JobUtilities.makeJob(JobUtilities.java:122)
        at com.islandpacific.gui.server.purchasing.productionorder.ProductionOrderDS.startSubmitForApprovalJob(ProductionOrderDS.java:1863)
        at com.islandpacific.gui.server.purchasing.productionorder.ProductionOrderDS.checkOutOfBalanceReq(ProductionOrderDS.java:941)
        at com.islandpacific.gui.server.purchasing.productionorder.ProductionOrderDS.submitForInitialApproval(ProductionOrderDS.java:727)
        at com.islandpacific.gui.server.purchasing.productionorder.ProductionOrderDS.executeUpdate(ProductionOrderDS.java:514)
        at com.isomorphic.datasource.DataSource.execute(DataSource.java:1967)
        at com.islandpacific.gui.server.customDataSource.IpDataSource.execute(IpDataSource.java:337)
        at com.isomorphic.application.AppBase.executeDefaultDSOperation(AppBase.java:726)
        at com.isomorphic.application.AppBase.executeAppOperation(AppBase.java:658)
        at com.isomorphic.application.AppBase.execute(AppBase.java:491)
        at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:2548)
        at com.isomorphic.servlet.IDACall.handleDSRequest(IDACall.java:220)
        at com.islandpacific.gui.server.customDataSource.IpIDACall.handleDSRequest(IpIDACall.java:113)
        at com.isomorphic.servlet.IDACall.processRPCTransaction(IDACall.java:185)
        at com.islandpacific.gui.server.customDataSource.IpIDACall.processRPCTransaction(IpIDACall.java:66)
        at com.isomorphic.servlet.IDACall.processRequest(IDACall.java:152)
        at com.isomorphic.servlet.IDACall._processRequest(IDACall.java:117)
        at com.isomorphic.servlet.IDACall.doPost(IDACall.java:76)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
        at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:156)


"http-bio-8088-exec-69" - Thread t@368
  java.lang.Thread.State: BLOCKED
"http-bio-8088-exec-69" - Thread t@368
  java.lang.Thread.State: BLOCKED
        at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:125)
        - waiting to lock <4abd797a> (a java.lang.Class) owned by "http-bio-8088-exec-68" t@367
        at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:86)
        at com.isomorphic.datasource.DSRequest.getDataSource(DSRequest.java:2216)
        at com.isomorphic.datasource.DSRequest.buildFieldData(DSRequest.java:3046)
        at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:2392)
        at com.isomorphic.datasource.DataSource.fetchById(DataSource.java:5319)
        at com.isomorphic.velocity.StoredRecordHandler.fetchStoredRecord(StoredRecordHandler.java:188)
        at com.isomorphic.velocity.StoredRecordHandler.size(StoredRecordHandler.java:232)
        at java.util.HashMap.<init>(HashMap.java:318)
        at com.isomorphic.velocity.EditedRecordHandler.prepareCombinedRecord(EditedRecordHandler.java:91)
        at com.isomorphic.velocity.EditedRecordHandler.put(EditedRecordHandler.java:177)
        at com.isomorphic.datasource.ValidationContext.setResultingValue(ValidationContext.java:210)
        at com.isomorphic.util.DefaultValidators.validateField(DefaultValidators.java:180)
        at com.isomorphic.datasource.SimpleType.validateValue(SimpleType.java:97)
        - locked <42d5dfc4> (a com.isomorphic.datasource.SimpleType)
        at com.isomorphic.datasource.SimpleType.create(SimpleType.java:73)
        at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:1962)
        at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:1877)
        at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:1742)
        at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1225)
        at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1209)
        at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1170)
        at com.isomorphic.datasource.DataSource.create(DataSource.java:1686)
        at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:1962)
        at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:1877)
        at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:1665)
        at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1225)
        at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1184)
        at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:1601)
        at com.isomorphic.xml.XML.toDSRecords(XML.java:363)
        at com.isomorphic.xml.XML.toDSRecords(XML.java:349)
        at com.isomorphic.xml.XML.toDSRecords(XML.java:345)
        at com.isomorphic.xml.XML.toDSRecords(XML.java:328)
        at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:966)
        at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:950)
        at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:919)
        at com.islandpacific.gui.server.customDataSource.IpDynamicDataSource.getIPDataSource(IpDynamicDataSource.java:121)
        at com.islandpacific.gui.server.customDataSource.IpDynamicDataSource.getDataSource(IpDynamicDataSource.java:106)
        at com.islandpacific.gui.server.customDataSource.IslandPacificDSLoader.processRequest(IslandPacificDSLoader.java:92)
        at com.isomorphic.servlet.DataSourceLoader.doPost(DataSourceLoader.java:101)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
        at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:156)


"http-bio-8088-exec-92" - Thread t@409
  java.lang.Thread.State: BLOCKED
        at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:125)
        - waiting to lock <4abd797a> (a java.lang.Class) owned by "http-bio-8088-exec-68" t@367
        at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:306)
        at com.isomorphic.datasource.BasicDataSource.findDataSource(BasicDataSource.java:2561)
        at com.isomorphic.datasource.BasicDataSource.getType(BasicDataSource.java:2447)
        at com.isomorphic.datasource.BasicDataSource.getFieldType(BasicDataSource.java:2198)
        at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:1893)
        at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:1877)
        at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:1665)
        at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1225)
        at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1184)
        at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:1601)
        at com.isomorphic.xml.XML.toDSRecords(XML.java:363)
        at com.isomorphic.xml.XML.toDSRecords(XML.java:349)
        at com.isomorphic.xml.XML.toDSRecords(XML.java:345)
        at com.isomorphic.xml.XML.toDSRecords(XML.java:328)
        at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:966)
        at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:950)
        at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:919)
        at com.islandpacific.gui.server.customDataSource.IpDynamicDataSource.getIPDataSource(IpDynamicDataSource.java:121)
        at com.islandpacific.gui.server.customDataSource.IpDynamicDataSource.getDataSource(IpDynamicDataSource.java:106)
        at com.islandpacific.gui.server.customDataSource.IslandPacificDSLoader.processRequest(IslandPacificDSLoader.java:92)
        at com.isomorphic.servlet.DataSourceLoader.doPost(DataSourceLoader.java:101)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
        at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:156)


"http-bio-8088-exec-94" - Thread t@426
  java.lang.Thread.State: BLOCKED
        at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:125)
        - waiting to lock <4abd797a> (a java.lang.Class) owned by "http-bio-8088-exec-68" t@367
        at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:86)
        at com.isomorphic.datasource.DSRequest.getDataSource(DSRequest.java:2216)
        at com.isomorphic.datasource.DSRequest.buildFieldData(DSRequest.java:3046)
        at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:2392)
        at com.islandpacific.gui.security.domain.IpApplicationManager.findAll(IpApplicationManager.java:85)
        at com.islandpacific.gui.security.IpAuthenticationServiceImpl.getAllApplications(IpAuthenticationServiceImpl.java:114)
        at com.islandpacific.gui.security.IpAuthenticationServiceImpl.getUserDetails(IpAuthenticationServiceImpl.java:67)
        at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:561)
        at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:265)
        at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:305)
        at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)

"http-bio-8088-exec-95" - Thread t@428
  java.lang.Thread.State: BLOCKED
        at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:125)
        - waiting to lock <4abd797a> (a java.lang.Class) owned by "http-bio-8088-exec-68" t@367
        at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:86)
        at com.isomorphic.datasource.DSRequest.getDataSource(DSRequest.java:2216)
        at com.isomorphic.datasource.DSRequest.buildFieldData(DSRequest.java:3046)
        at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:2392)
        at com.islandpacific.gui.security.domain.IpApplicationManager.findAll(IpApplicationManager.java:85)
        at com.islandpacific.gui.security.IpAuthenticationServiceImpl.getAllApplications(IpAuthenticationServiceImpl.java:114)
        at com.islandpacific.gui.security.IpAuthenticationServiceImpl.getUserDetails(IpAuthenticationServiceImpl.java:67)
        at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:561)
        at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:265)
        at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:305)
        at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)

So, threads 66 and 68 are waiting for a lock <42d5dfc4> to be released by thread 69 and threads 69, 92, 94 and 95 are all waiting on thread 68 to release it's lock <4abd797a>.

Threads 66 and 68 are both within SimpleType.validateValue() which seems to have been synchronized for a long time. The other threads are within getDataSource(), which a decompiler shows me is now synchronized:

[EDIT: I notice the EULA forbids me from decompiling the code, so I will list only the method signatures below and apologize in advance if I have broken the license terms by looking at the underlying code.]

Code:

  public static synchronized DataSource getDataSource(String name, DSRequest dsRequest, boolean isSchemaless)
...whereas it didn't used to be (from v8.3p_2014-08-22)...

Code:

  public static DataSource getDataSource(String name, DSRequest dsRequest, boolean isSchemaless)

We are hoping that something in here allows you to either suggest what we might look at in our code, or alternatively that we might have hit some kind of edge case failure that you are able to address. Thank you for your time.

Viewing all articles
Browse latest Browse all 4756

Trending Articles