"Session is closed!" Message

pilotstevek · September 30, 2009, 11:18am

I have some pages being published with this error message displayed instead of their content

Problem assembling output for item: 16-101-4866 with template: erauPgGeneric exception: Session is closed! see log for stack trace

The console log shows:

2009-09-30 09:55:34,386 ERROR [PSVelocityAssembler] Problem assembling output for item: 16-101-4866 with template: erauPgGeneric
org.hibernate.SessionException: Session is closed!
	at org.hibernate.impl.AbstractSessionImpl.errorIfClosed(AbstractSessionImpl.java:49)
	at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:993)
	at org.springframework.orm.hibernate3.SessionFactoryUtils$SpringSessionSynchronization.beforeCommit(SessionFactoryUtils.java:884)
	at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:48)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:707)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:523)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:510)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:310)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:117)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:209)
	at $Proxy90.assemble(Unknown Source)
	at com.percussion.services.assembly.jexl.PSAssemblerUtils.assemble(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor216.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:267)
	at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:197)
	at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:175)
	at org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:327)
	at org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:51)
	at org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:95)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:55)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:70)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:230)
	at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:172)
	at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:114)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:230)
	at org.apache.velocity.Template.merge(Template.java:256)
	at com.percussion.services.assembly.impl.plugin.PSVelocityAssembler.assembleSingle(Unknown Source)
	at com.percussion.services.assembly.impl.plugin.PSAssemblerBase.doAssembleSingle(Unknown Source)
	at com.percussion.services.assembly.impl.plugin.PSVelocityAssembler.doAssembleSingle(Unknown Source)
	at com.percussion.services.assembly.impl.plugin.PSAssemblerBase$_b.o00000(Unknown Source)
	at com.percussion.services.assembly.impl.plugin.PSAssemblerBase$_b.call(Unknown Source)
	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at com.percussion.services.assembly.impl.plugin.PSAssemblerBase.assemble(Unknown Source)
	at com.percussion.services.assembly.impl.PSAssemblyService.assemble(Unknown Source)
	at com.percussion.services.assembly.impl.PSAssemblyService.processServletRequest(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor379.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
	at $Proxy90.processServletRequest(Unknown Source)
	at com.percussion.services.assembly.impl.PSAssemblyServlet.service(Unknown Source)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
	at com.percussion.webdav.PSWebDavRequestFilter.doFilter(Unknown Source)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
	at com.percussion.servlets.PSSecurityFilter.doFilter(Unknown Source)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
	at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
	at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
	at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:159)
	at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
	at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
	at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
	at java.lang.Thread.run(Unknown Source)

Sometimes there is an entry like this:


2009-09-29 14:02:06,417 INFO  [Server] Shutting down psx_ceerauNews
2009-09-29 14:02:07,102 INFO  [Server] Shutting down .sys_CEHandler2599
2009-09-29 14:02:07,116 INFO  [Server] Initializing application: psx_ceerauNews
2009-09-29 14:02:07,163 ERROR [PSAssemblerUtils] Problem during assembly
org.hibernate.SessionException: Session is closed!
	at org.hibernate.impl.AbstractSessionImpl.errorIfClosed(AbstractSessionImpl.java:49)
	at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:993)
	at org.springframework.orm.hibernate3.SessionFactoryUtils$SpringSessionSynchronization.beforeCommit(SessionFactoryUtils.java:884)
	at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:48)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:707)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:523)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:510)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:310)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:117)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:209)
	at $Proxy90.assemble(Unknown Source)
	at com.percussion.services.assembly.jexl.PSAssemblerUtils.assemble(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor216.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:267)
	at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:197)
	at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:175)
	at org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:327)
	at org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:51)
	at org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:95)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:55)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:70)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:230)
	at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:172)
	at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:114)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:230)
	at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:172)
	at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:114)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:230)
	at org.apache.velocity.Template.merge(Template.java:256)
	at com.percussion.services.assembly.impl.plugin.PSVelocityAssembler.assembleSingle(Unknown Source)
	at com.percussion.services.assembly.impl.plugin.PSAssemblerBase.doAssembleSingle(Unknown Source)
	at com.percussion.services.assembly.impl.plugin.PSVelocityAssembler.doAssembleSingle(Unknown Source)
	at com.percussion.services.assembly.impl.plugin.PSAssemblerBase$_b.o00000(Unknown Source)
	at com.percussion.services.assembly.impl.plugin.PSAssemblerBase$_b.call(Unknown Source)
	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at com.percussion.services.assembly.impl.plugin.PSAssemblerBase.assemble(Unknown Source)
	at com.percussion.services.assembly.impl.PSAssemblyService.assemble(Unknown Source)
	at com.percussion.services.assembly.impl.PSAssemblyService.processServletRequest(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor379.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
	at $Proxy90.processServletRequest(Unknown Source)
	at com.percussion.services.assembly.impl.PSAssemblyServlet.service(Unknown Source)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
	at com.percussion.webdav.PSWebDavRequestFilter.doFilter(Unknown Source)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
	at com.percussion.servlets.PSSecurityFilter.doFilter(Unknown Source)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
	at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
	at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
	at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:159)
	at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
	at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
	at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
	at java.lang.Thread.run(Unknown Source)

The problem is isolated to today, the content types being published with the error are random, so far it has been seen on a generic, news, and a center content type.

There seems to be some weird duplicate processes (cqquery_, cqsched_,cqserv_) running on the box that I haven’t seen before:

UID        PID  PPID  C STIME TTY          TIME CMD
tcuser     444 25431  0 10:02 ?        00:00:00 cqquery_1 -library ce101 ce101;229 ce101;435 ce301 ce302 ce303 ce305 c
tcuser     447 25431  0 10:02 ?        00:00:00 cqsched_1 -cfg /cms/Rhythmyx/sys_search/rware/rx/config/rware.cfg -que
tcuser     449 25431  0 10:02 ?        00:00:00 cqserv_1 -library ce101 ce101;229 ce101;435 ce301 ce302 ce303 ce305 ce
tcuser     451 25431  0 10:02 ?        00:00:00 cqdh -no_hist -error_log /cms/Rhythmyx/sys_search/rware/rx/logs/cqdh.e
tcuser     453 25431  0 10:02 ?        00:00:00 cqindex_ce508 -library ce508 -cfg /cms/Rhythmyx/sys_search/rware/rx/co
tcuser     455 25431  0 10:02 ?        00:00:00 cqxref -data_dir /cms/Rhythmyx/sys_search/rware/rx/indexes -error_log 
tcuser    3213 17322  0 11:07 ?        00:00:00 /local/apache/bin/httpd -k start -f /local/apache/conf/httpd.conf
tcuser    3741 17322  0 10:36 ?        00:00:00 /local/apache/bin/httpd -k start -f /local/apache/conf/httpd.conf
tcuser    4832 17322  0 11:08 ?        00:00:00 /local/apache/bin/httpd -k start -f /local/apache/conf/httpd.conf
tcuser    4859 17322  0 11:09 ?        00:00:00 /local/apache/bin/httpd -k start -f /local/apache/conf/httpd.conf
tcuser    4874 25431  0 11:09 ?        00:00:00 cqindex_ce324 -library ce324 -cfg /cms/Rhythmyx/sys_search/rware/rx/co
tcuser    6625 17322  0 11:10 ?        00:00:00 /local/apache/bin/httpd -k start -f /local/apache/conf/httpd.conf
tcuser    9368 17322  0 11:13 ?        00:00:00 /local/apache/bin/httpd -k start -f /local/apache/conf/httpd.conf
tcuser   10404 13307  0 11:14 pts/1    00:00:00 ps -fu tcuser
tcuser   13306 13304  0 08:42 ?        00:00:01 sshd: tcuser@pts/1
tcuser   13307 13306  0 08:42 pts/1    00:00:00 -bash
tcuser   13308 13306  0 08:42 ?        00:00:00 /usr/libexec/openssh/sftp-server
tcuser   16576     1  0 Sep22 ?        00:00:00 ./RhythmyxDaemon start ..
tcuser   16577 16576  0 Sep22 ?        00:00:00 sh -c sh ./StartServer.sh 1>/dev/null 2>/dev/null <&-
tcuser   16579 16577  0 Sep22 ?        00:00:00 sh ./StartServer.sh
tcuser   16593 16579  0 Sep22 ?        00:00:00 /bin/sh ./doStart.sh
tcuser   16596 16593  0 Sep22 ?        00:00:00 /cms/Rhythmyx/RhythmyxServer.bin
tcuser   16612 16596  0 Sep22 ?        00:00:00 /cms/Rhythmyx/RhythmyxServer.bin
tcuser   16613 16612 26 Sep22 ?        2-01:15:51 /cms/Rhythmyx/JRE/bin/java -cp :JRE/lib/tools.jar:AppServer/bin/run.
tcuser   21576 17322  0 10:23 ?        00:00:00 /local/apache/bin/httpd -k start -f /local/apache/conf/httpd.conf
tcuser   22649 17322  0 10:54 ?        00:00:00 /local/apache/bin/httpd -k start -f /local/apache/conf/httpd.conf
tcuser   25431 16613  0 09:55 ?        00:00:00 /cms/Rhythmyx/sys_search/rware/bin/execd -cfg /cms/Rhythmyx/sys_search
tcuser   25440 25431  0 09:55 ?        00:00:00 cqns -error_log /cms/Rhythmyx/sys_search/rware/rx/logs/cqns.err -lan_p
tcuser   27470 17322  0 10:59 ?        00:00:00 /local/apache/bin/httpd -k start -f /local/apache/conf/httpd.conf
tcuser   29246 17322  0 11:01 ?        00:00:00 /local/apache/bin/httpd -k start -f /local/apache/conf/httpd.conf
tcuser   29647 25431  0 10:30 ?        00:00:00 cqquery_4 -library ce334 ce335 ce340 ce340;2 ce344 ce346 ce358 ce368 c
tcuser   29649 25431  0 10:30 ?        00:00:00 cqsched_4 -cfg /cms/Rhythmyx/sys_search/rware/rx/config/rware.cfg -que
tcuser   29651 25431  0 10:30 ?        00:00:00 cqserv_4 -library ce334 ce335 ce340 ce340;2 ce344 ce346 ce358 ce368 ce
tcuser   29653 25431  0 10:30 ?        00:00:00 cqquery_3 -library ce321 ce322 ce323 ce324 ce325 ce326 ce327 ce330 ce3
tcuser   29655 25431  0 10:30 ?        00:00:00 cqsched_3 -cfg /cms/Rhythmyx/sys_search/rware/rx/config/rware.cfg -que
tcuser   29657 25431  0 10:30 ?        00:00:00 cqserv_3 -library ce321 ce322 ce323 ce324 ce325 ce326 ce327 ce330 ce33
tcuser   29665 25431  0 10:30 ?        00:00:00 cqquery_2 -library ce310 ce311 ce312 ce313 ce314 ce315 ce316 ce318 ce3
tcuser   29667 25431  0 10:30 ?        00:00:00 cqsched_2 -cfg /cms/Rhythmyx/sys_search/rware/rx/config/rware.cfg -que
tcuser   29669 25431  0 10:30 ?        00:00:00 cqserv_2 -library ce310 ce311 ce312 ce313 ce314 ce315 ce316 ce318 ce31
tcuser   29671 25431  0 10:31 ?        00:00:00 cqquery_6 -library ce499 ce508 -cfg /cms/Rhythmyx/sys_search/rware/rx/
tcuser   29673 25431  0 10:31 ?        00:00:00 cqsched_6 -cfg /cms/Rhythmyx/sys_search/rware/rx/config/rware.cfg -que
tcuser   29675 25431  0 10:31 ?        00:00:00 cqserv_6 -library ce499 ce508 -cfg /cms/Rhythmyx/sys_search/rware/rx/c
tcuser   29677 25431  0 10:31 ?        00:00:00 cqquery_5 -library ce391 ce408 ce418 ce428 ce438 ce448 ce468 ce478 ce4
tcuser   29679 25431  0 10:31 ?        00:00:00 cqsched_5 -cfg /cms/Rhythmyx/sys_search/rware/rx/config/rware.cfg -que
tcuser   29685 25431  0 10:31 ?        00:00:00 cqserv_5 -library ce391 ce408 ce418 ce428 ce438 ce448 ce468 ce478 ce48
tcuser   31764 25431  0 10:32 ?        00:00:00 cqindex_ce438 -library ce438 -cfg /cms/Rhythmyx/sys_search/rware/rx/co

Has anyone seen this before?

bwilder · September 30, 2009, 1:20pm

The cq* processes are all related to the Convera FTS. Have you tried restarting Rx?

pilotstevek · October 1, 2009, 9:42am

We have had similar problems with these cq processes building up before which eventually caused the search engine to become unresponsive. Restarting the server cleared this problem up too. Could this be a performance issue with the server? This will be the 3rd time I have needed to restart our production server due to this type of behavior. Should I open a work order for some type of load testing with our configuration?

bwilder · October 1, 2009, 10:09am

If you’ve had this issue before, it’s probably Convera. Maybe 1 or more of your indexes are corrupted OR a field on a content type that FTS is enabled on is corrupted OR it could be something else entirely. The easiest thing to do would be to disable Convera entirely. Doing so will remove the large “Search For” box in the top of the search dialog, but you will still be able to search all other fields (contentid, title, community, etc). Alternatively, you may want try doing a full reindex, which may take several hours, but may resolve the issue without disabling the FTS feature.

Turning off Convera FTS is a simple process:

[ol]
[li]Shut down your Rx server,[/li][li]Back up <rxroot>/rxconfig/Server/config.xml,</rxroot>[/li][li]Open config.xml and sesarch for “fullTextSearchEnabled”,[/li][li]Change the value of this attribute from “yes” to “no”,[/li][li]Also change the serverPort attribute from “9993” (your APORT) to “-1”,[/li][li]Save, Restart Rx[/li][/ol]
If you’d like to debug your environment further, I’d recommend opening a ticket with TS.