Unable to Access Percussion

We’re experiencing a problem with access to Percussion that went down this afternoon unexpectedly.

On the login screen, we get the following error:

HTTP Status 404 - /Rhythmyx/login

type Status report

message /Rhythmyx/login

description The requested resource (/Rhythmyx/login) is not available.

We’ve tried restarting the server, but without any luck.

Nix that! After another server restart, it’s back up!

Matthew,

How much memory is available on the machine that you are running CM1? It could be that your memory settings just need updated from the defaults.

-n

Nate,

I’m not sure. How would I check memory? We don’t have visual access of the server.

Percussion just went down again with an error:

2012-08-29 16:27:11,073 ERROR [com.percussion.sitemanage.servlet.PSPreviewItemContent] com.percussion.share.service.IPSDataService$DataServiceLoadException: com.percussion.webservices.PSErrorException: Failed to load objects of type ‘Could not commit Hibernate transaction; nested exception is org.hibernate.TransactionException: JDBC commit failed’ with the criteria (filter or ids) of ‘{1}’. The underlying error was: {2}.

Matt,

I’m glad to hear that. I’m going to edit out your log text above now that this has been resolved. For customers with this issue in the future, I believe what was going on here was a port conflict, which typically occurs when you try to restart CM1 and the prior instance of CM1 never shut down fully. If this is the case, you will face an error through the browser similar to what Matt posted, and you will see an error similar to this in your log:

“java.net.ConnectException : Error connecting to server localhost on port 1527 with message Connection refused: connect.”

One resolution, obviously, is to restart the machine running CM1. If that’s not an easy option, however, there are two ways to ensure that CM1 has shutdown completely before starting back.

In Windows:

Stop the percussion service, download Process Explorer here: http://technet.microsoft.com/en-us/sy… (you can use Task Manager also, but this will be easier), and kill the whole process tree running off of PercussionCM.exe (all the “java.exe” processes, primarily). Then you should be able to restart the Percussion service without conflict.

In Linux:

Stop the percussion process. Use the command “ps -ef | grep java” to display all the java processes running in that machine. Kill all of the processes running out of the Percussion directory using “kill -9” followed by the numerical ID of the process. Then start the Percussion service back up once all the Percussion java processes have been shut down.

Nathaniel

Oh, now I see you have encountered another error. The above should still be relevant to other customers faced with the initial error in this topic.

Matt,

Are you unable to hit the CM1 user interface following this new error being thrown?

Nathaniel

Initially after the error, no. I was getting Status 500 error on pages.

Restarting Percussion again fixed the problem for now. But I wonder what’s causing it to shut down like that.

I’ll let you know if it continues to happen.

Very odd. Please let me know if anything like this happens again, and I will create an internal support ticket for us to look into potential causes of these shutdowns.

We just got in this morning to find Percussion down and this error in our log:

2012-08-30 08:12:39,177 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/cm].[jsp]] Servlet.service() for servlet jsp threw exception
java.io.IOException: Server returned HTTP response code: 500 for URL: http://127.0.0.1:9992/Rhythmyx/servic…

Looking back (I had the log open), there’s an error every so often that look like this:

2012-08-29 16:20:30,160 ERROR [com.percussion.sitemanage.servlet.PSPreviewItemContent] com.percussion.share.service.IPSDataService$DataServiceLoadException: com.percussion.webservices.PSErrorException: Failed to load objects of type ‘Could not commit Hibernate transaction; nested exception is org.hibernate.TransactionException: JDBC commit failed’ with the criteria (filter or ids) of ‘{1}’. The underlying error was: {2}.
2012-08-29 16:20:57,357 INFO [com.percussion.Aging Agent] Polling aging action at: Wed Aug 29 16:20:57 EDT 2012…
2012-08-29 16:20:57,359 ERROR [com.percussion.data.ub] Application sys_ageSupport, Dataset agecontentlist, Request agecontentlist

This has been resolved by Nick through a support ticket. Turns out, we ran out of space on our petition, which was causing Percussion to go down. After removing some unneeded files (CM1 2.7 install, for example), we’re good to go for now.

May look into increasing the petition size (4 gigs currently) in the future, since our site contains a lot of files–mostly images.

I helped make the following memory changes for derby in cm1 which installed on a Linux server:

The first ist located in the DataBaseStartup.sh script located in [Cm1Root]

@echo off   
cd JRE64\bin   
java -server -Xss2048k -Xms1024m -Xmx1024m -XX:MaxPermSize=128M   

I added in the following:

-server -Xss2048k -Xms1024m

Next I updated derby.properties located in [Cm1Root]\Repository

derby.storage.pageSize=32768   
derby.storage.pageCacheSize=2000   
derby.language.statementCacheSize=1000 ```   
 
I finally updated rx-ds.xml located here **[CM1Root]\AppServer\server\rx\deploy**   
 
<?xml version="1.0" encoding="utf-8"?>





jdbc/RhythmyxData

jdbc:derby://localhost:1527/CMDB;

org.apache.derby.jdbc.ClientDriver

TRANSACTION_READ_UNCOMMITTED

CMDB

demo

5

100

15



DERBY




Looking into this I found that the server was out of disk space on the file system.

Also in the server.log file I noticed the following message:

The call-stack was:   
com.percussion.search.PSSearchException:   
at com.percussion.search.lucene.PSSearchIndexerImpl.update(Unknown Source)   
at com.percussion.search.PSSearchIndexEventQueue.indexItemChanges(Unknown Source)   
at com.percussion.search.PSSearchIndexEventQueue.processNextEventSet(Unknown Source)   
at com.percussion.search.PSSearchIndexEventQueue.access$100(Unknown Source)   
at com.percussion.search.PSSearchIndexEventQueue$1.run(Unknown Source)   
Caused by: java.io.IOException: No space left on device  

the line Caused by: java.io.IOException: No space left on device

Indicates that there is no space left on the mount point

When I ran the following command:

df -h

I get the following output:

4.0G 4.0G 84K 100% /home