Discussion:
prod env problem!
(too old to reply)
HARPREET SINGH
2005-02-07 23:48:38 UTC
Permalink
Hi,
We have Weblogic 8.1 on solaris platform and have a critical application one managed server.This application/server is hangs up quite often ,like 4,5 times in a week.There are 400 execute threads and 3 reader threads.Most of the time the execute threads remain idle ,at the time of problem too, and reader thread are busy.
I get following Exceptions in log files at different time of problems:
####<Feb 7, 2005 8:22:30 AM CST> <Info> <EJB> <lcy002194> <production> <ExecuteThread: '389' for queue: 'weblogic.kernel.Default'> <<anonymous>> <> <BEA-010049> <EJB Exception in method: remove: java.rmi.NoSuchObjectException: Bean has been deleted..
java.rmi.NoSuchObjectException: Bean has been deleted.
at weblogic.ejb20.swap.DiskSwap.read(DiskSwap.java:172)
at weblogic.ejb20.manager.StatefulSessionManager.getBean(StatefulSessionManager.java:326)
at weblogic.ejb20.manager.StatefulSessionManager.preInvoke(StatefulSessionManager.java:396)
at weblogic.ejb20.manager.StatefulSessionManager.remove(StatefulSessionManager.java:936)
at weblogic.ejb20.manager.ReplicatedStatefulSessionManager.remove(ReplicatedStatefulSessionManager.java:196)
at weblogic.ejb20.internal.StatefulEJBObject.remove(StatefulEJBObject.java:92)
at com.granite.middle.session.ClientServiceEJB_u5heex_EOImpl.remove(ClientServiceEJB_u5heex_EOImpl.java:4642)
at com.granite.middle.session.ClientServiceEJB_u5heex_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:466)
at weblogic.rmi.cluster.ReplicaAwareServerRef.invoke(ReplicaAwareServerRef.java:108)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:409)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:353)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:144)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:404)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:30)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)
####<Feb 7, 2005 8:23:38 AM CST> <Info> <EJB> <lcy002194> <production> <ExecuteThread: '389' for queue: 'weblogic.kernel.Default'> <<anonymous>> <BEA1-14E3B0CB38DA> <BEA-010051> <EJB Exception occurred during invocation from home: ***@1ec56c9 threw exception: java.lang.NullPointerException
java.lang.NullPointerException
at com.granite.middle.worxng.session.QueueBean.getViewPrivilegeQueueList(QueueBean.java:1056)
at com.granite.middle.worxng.session.Queue_68j9s_EOImpl.getViewPrivilegeQueueList(Queue_68j9s_EOImpl.java:208)
at com.granite.middle.worxng.session.Queue_68j9s_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:466)
at weblogic.rmi.cluster.ReplicaAwareServerRef.invoke(ReplicaAwareServerRef.java:108)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:409)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:353)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:144)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:404)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:30)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java


####<Feb 7, 2005 4:50:20 PM CST> <Info> <EJB> <lcy002194> <production> <ExecuteThread: '390' for queue: 'weblogic.kernel.Default'> <<anonymous>> <BEA1-0371ACF1523A94FFBE2B> <BEA-010051> <EJB Exception occurred during invocation from home: ***@f5c440 threw exception: weblogic.utils.NestedError: Unexpected Exception: - with nested exception:
[java.security.PrivilegedActionException: java.sql.SQLException: Transaction rolled back: Transaction timed out after 599 seconds
Xid=BEA1-0371ACF1523A94FFBE2B(23844485),Status=Active,numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=599,seconds left=60,activeThread=Thread[ExecuteThread: '390' for queue: 'weblogic.kernel.Default',5,Thread Group for Queue: 'weblogic.kernel.Default'],XAServerResourceInfo[JMS_xngJMSFileStore]=(ServerResourceInfo[JMS_xngJMSFileStore]=(state=suspended,assigned=none),xar=JMS_xngJMSFileStore),XAServerResourceInfo[xngprod1_prod]=(ServerResourceInfo[xngprod1_prod]=(state=started,assigned=none),xar=xngprod1_prod),SCInfo[granite+production]=(state=active),local properties=({weblogic.jdbc.jta.xngprod1_prod=***@d0c4da}),OwnerTransactionManager=ServerTM[ServerCoordinatorDescriptor=(CoordinatorURL=production+10.174.2.194:7280+granite+t3+, XAResources={},NonXAResources={})],CoordinatorURL=production+10.174.2.194:7280+granite+t3+)]
java.security.PrivilegedActionException: java.sql.SQLException: Transaction rolled back: Transaction timed out after 599 seconds
Xid=BEA1-0371ACF1523A94FFBE2B(23844485),Status=Active,numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=599,seconds left=60,activeThread=Thread[ExecuteThread: '390' for queue: 'weblogic.kernel.Default',5,Thread Group for Queue: 'weblogic.kernel.Default'],XAServerResourceInfo[JMS_xngJMSFileStore]=(ServerResourceInfo[JMS_xngJMSFileStore]=(state=suspended,assigned=none),xar=JMS_xngJMSFileStore),XAServerResourceInfo[xngprod1_prod]=(ServerResourceInfo[xngprod1_prod]=(state=started,assigned=none),xar=xngprod1_prod),SCInfo[granite+production]=(state=active),local properties=({weblogic.jdbc.jta.xngprod1_prod=***@d0c4da}),OwnerTransactionManager=ServerTM[ServerCoordinatorDescriptor=(CoordinatorURL=production+10.174.2.194:7280+granite+t3+, XAResources={},NonXAResources={})],CoordinatorURL=production+10.174.2.194:7280+granite+t3+)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:144)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:188)
at com.granite.middle.helper.jdbc.ConnectionJDBC.getNewConnection(ConnectionJDBC.java:843)
at com.granite.middle.helper.jdbc.ConnectionJDBC.getConnection(ConnectionJDBC.java:819)
at com.granite.middle.helper.jdbc.ProfilePrivilegeHelper.checkPrivilege(ProfilePrivilegeHelper.java:553)
at com.granite.middle.helper.jdbc.ProfilePrivilegeHelper.checkPrivilege(ProfilePrivilegeHelper.java:452)
at com.granite.middle.logic.PathLogic.getItem(PathLogic.java:429)
at com.granite.middle.session.PathBean.getItem(PathBean.java:834)
at com.granite.middle.session.PathBean.getItem(PathBean.java:809)
at com.granite.middle.session.PathEJB_9q5d2l_EOImpl.getItem(PathEJB_9q5d2l_EOImpl.java:2854)
at com.granite.bdl.protocol.IIOP.adapter.IIOPPathAdapter.load(IIOPPathAdapter.java:246)
at com.granite.bdl.protocol.IIOP.adapter.IIOPPathAdapter.loadTransport(IIOPPathAdapter.java:1700)
at com.granite.bdl.protocol.BDLAdapterImpl.load(BDLAdapterImpl.java:69)
at com.granite.bdl.protocol.PathImpl.load(PathImpl.java:4656)
at com.granite.bdl.protocol.PathImpl.loadLiveArchive(PathImpl.java:5004)
at com.granite.xng_events.DefaultXngEventSerializer.getEventElement(DefaultXngEventSerializer.java:442)
at com.granite.xng_events.DefaultXngEventSerializer.serialize(DefaultXngEventSerializer.java:92)
at com.granite.xng_events.XngEventMessageBean.onMessage(XngEventMessageBean.java:442)
at weblogic.ejb20.internal.MDListener.execute(MDListener.java:382)
at weblogic.ejb20.internal.MDListener.transactionalOnMessage(MDListener.java:316)
at weblogic.ejb20.internal.MDListener.onMessage(MDListener.java:281)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:2596)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:2516)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)
Caused by: java.sql.SQLException: Transaction rolled back: Transaction timed out after 599 seconds
Xid=BEA1-0371ACF1523A94FFBE2B(23844485),Status=Active,numRepliesOwedMe=0,numRepliesOwedOthers=0,second



Please provide me some ideas!
Thanks
Rob Woollen
2005-02-08 00:39:29 UTC
Permalink
Post by HARPREET SINGH
Hi,
We have Weblogic 8.1 on solaris platform and have a critical application one managed server.This application/server is hangs up quite often ,like 4,5 times in a week.There are 400 execute threads and 3 reader threads.Most of the time the execute threads remain idle ,at the time of problem too, and reader thread are busy.
####<Feb 7, 2005 8:22:30 AM CST> <Info> <EJB> <lcy002194> <production> <ExecuteThread: '389' for queue: 'weblogic.kernel.Default'> <<anonymous>> <> <BEA-010049> <EJB Exception in method: remove: java.rmi.NoSuchObjectException: Bean has been deleted..
java.rmi.NoSuchObjectException: Bean has been deleted.
at weblogic.ejb20.swap.DiskSwap.read(DiskSwap.java:172)
at weblogic.ejb20.manager.StatefulSessionManager.getBean(StatefulSessionManager.java:326)
at weblogic.ejb20.manager.StatefulSessionManager.preInvoke(StatefulSessionManager.java:396)
at weblogic.ejb20.manager.StatefulSessionManager.remove(StatefulSessionManager.java:936)
at weblogic.ejb20.manager.ReplicatedStatefulSessionManager.remove(ReplicatedStatefulSessionManager.java:196)
at weblogic.ejb20.internal.StatefulEJBObject.remove(StatefulEJBObject.java:92)
at com.granite.middle.session.ClientServiceEJB_u5heex_EOImpl.remove(ClientServiceEJB_u5heex_EOImpl.java:4642)
at com.granite.middle.session.ClientServiceEJB_u5heex_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:466)
at weblogic.rmi.cluster.ReplicaAwareServerRef.invoke(ReplicaAwareServerRef.java:108)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:409)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:353)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:144)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:404)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:30)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)
This is a remove of a stateful session bean which has already been
deleted. Should be harmless.
Post by HARPREET SINGH
java.lang.NullPointerException
at com.granite.middle.worxng.session.QueueBean.getViewPrivilegeQueueList(QueueBean.java:1056)
at com.granite.middle.worxng.session.Queue_68j9s_EOImpl.getViewPrivilegeQueueList(Queue_68j9s_EOImpl.java:208)
at com.granite.middle.worxng.session.Queue_68j9s_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:466)
at weblogic.rmi.cluster.ReplicaAwareServerRef.invoke(ReplicaAwareServerRef.java:108)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:409)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:353)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:144)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:404)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:30)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java
This is a NullPointerException in some com.granite.* code. That appears
to be a bug in that code.
Post by HARPREET SINGH
[java.security.PrivilegedActionException: java.sql.SQLException: Transaction rolled back: Transaction timed out after 599 seconds
java.security.PrivilegedActionException: java.sql.SQLException: Transaction rolled back: Transaction timed out after 599 seconds
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:144)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:188)
at com.granite.middle.helper.jdbc.ConnectionJDBC.getNewConnection(ConnectionJDBC.java:843)
at com.granite.middle.helper.jdbc.ConnectionJDBC.getConnection(ConnectionJDBC.java:819)
at com.granite.middle.helper.jdbc.ProfilePrivilegeHelper.checkPrivilege(ProfilePrivilegeHelper.java:553)
at com.granite.middle.helper.jdbc.ProfilePrivilegeHelper.checkPrivilege(ProfilePrivilegeHelper.java:452)
at com.granite.middle.logic.PathLogic.getItem(PathLogic.java:429)
at com.granite.middle.session.PathBean.getItem(PathBean.java:834)
at com.granite.middle.session.PathBean.getItem(PathBean.java:809)
at com.granite.middle.session.PathEJB_9q5d2l_EOImpl.getItem(PathEJB_9q5d2l_EOImpl.java:2854)
at com.granite.bdl.protocol.IIOP.adapter.IIOPPathAdapter.load(IIOPPathAdapter.java:246)
at com.granite.bdl.protocol.IIOP.adapter.IIOPPathAdapter.loadTransport(IIOPPathAdapter.java:1700)
at com.granite.bdl.protocol.BDLAdapterImpl.load(BDLAdapterImpl.java:69)
at com.granite.bdl.protocol.PathImpl.load(PathImpl.java:4656)
at com.granite.bdl.protocol.PathImpl.loadLiveArchive(PathImpl.java:5004)
at com.granite.xng_events.DefaultXngEventSerializer.getEventElement(DefaultXngEventSerializer.java:442)
at com.granite.xng_events.DefaultXngEventSerializer.serialize(DefaultXngEventSerializer.java:92)
at com.granite.xng_events.XngEventMessageBean.onMessage(XngEventMessageBean.java:442)
at weblogic.ejb20.internal.MDListener.execute(MDListener.java:382)
at weblogic.ejb20.internal.MDListener.transactionalOnMessage(MDListener.java:316)
at weblogic.ejb20.internal.MDListener.onMessage(MDListener.java:281)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:2596)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:2516)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)
Caused by: java.sql.SQLException: Transaction rolled back: Transaction timed out after 599 seconds
Xid=BEA1-0371ACF1523A94FFBE2B(23844485),Status=Active,numRepliesOwedMe=0,numRepliesOwedOthers=0,second
This is a transaction which has timedout after 10 minutes. First off,
10 minutes is too long for your tx timeout. I would not recommend
increasing the timeout beyond the default (30 seconds).

I'd suggest monitoring the server and taking some thread dumps when you
notice the problems. I suspect something is hung.

-- Rob
Post by HARPREET SINGH
Please provide me some ideas!
Thanks
Loading...