Matt,
I understand your team uses Macs. I am currently running some tests to see if any of your issues here and in your two other posts are Mac-specific. I’ll get back to you shortly to let you know what I find. For this problem in particular it might have to do with your database, in which case I can open up a support ticket for you to look into this deeper. Do you know if you’re using the embedded Derby database, or an external SQL database?
Nathaniel
Nathaniel,
We’re using the embedded Derby database.
Thanks.
I wanted to add that this morning’s slow down seems different in nature. We see the effect when logging in and when we open folders. The root level itself shows the spinning circle for some time before showing its contents.
Matt,
Thanks for the information. Can you give me a rough estimate of the amount of pages and users in your system? I’m looking into whether the types of slowdowns you are experiencing are consistent with the issues you might face when operating a large Derby database, or if this is an unrelated issue.
Nathaniel
We currently have 931 pages named index and roughly 10 users–3 of which will be working within Percussion consistently.
We did stop/start Percussion, and it seemed to run faster immediately after logging in.
Matt,
I’m glad to hear the restart appears to have sped you system up. Another thing to check is if you use any combination of the Activity, Traffic, and What’s Working gadgets on you dashboard, these can use a large amount of database bandwidth, which might lead to the slowdowns you’ve described. If you are using any of these gadgets and encounter further slow response times, try disabling them and see if that speeds things up.
Nathaniel
Nathaniel,
I don’t think any of our users are using any combinations of those widgets.
We came back from the weekend to find it running slowly again–to the point of unresponsive. After performing a server restart, it was immediately fixed (for the time being).
I wonder what’s causing the gradual decline in response time?
Matt,
It does sound like a memory issue, then. Can you tell me if you are on CM1 2.6 or 2.7? There was a known inefficacy with the Derby database which affected 2.6, leading to memory usage issues and slowdowns similar to what you’re describing. If memory serves, however, we were on a call with you earlier this month to upgrade to 2.7. Is this correct?
Another thing, do you know if your CM1 application server is running in a 32bit or 64bit environment?
I’m looking into other possible ways to improve your performance; I will get back to you when I know more.
Nathaniel
Nathaniel,
We’re using version 2.7 of CM1, and we’re running it on a server with Ubuntu 10.04 LTS minimal system (64-bit).
Matthew,
You can check the available memory on your Unbuntu server using the “free -t -m” command. It will list totals in megabytes. It would also be usefull to see the output from top to see how much memory java processes are using.
There are 2 files that contain settings that control how much memory CM1 uses on Linux with embedded Derby both in the installation directory.
1.) PercussionServer.bin.lax
The line:
lax.nl.java.option.additional=-Dprogram.name=RhythmyxServer.exe -Xms128m -Xmx1024m -Djava.endorsed.dirs=“AppServer/lib/endorsed” -Djava.library.path=./bin -server -XX:MaxPermSize=128m -Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF8
By default this is set to 1024m (1 GB) of memory.
2.) DatabaseStartup.sh
The line:
./java -server -Xms512m -Xmx512m -XX:MaxPermSize=128M -classpath …/…/AppServer/server/rx/lib/rxlogin.jar:…/…/AppServer/server/rx/deploy/rxapp.ear/rxapp.war/WEB-INF/lib/rxclient.jar:…/…/AppServer/server/rx/deploy/rxapp.ear/rxapp.war/WEB-INF/lib/rxutils.jar:…/…/AppServer/server/rx/deploy/rxapp.ear/rxapp.war/WEB-INF/lib/rxserver.jar:…/…/AppServer/server/rx/deploy/rxapp.ear/rxapp.war/WEB-INF/lib/commons-lang-2.4.jar:…/…/Repository/lib/derbynet.jar -Dderby.system.home=…/…/Repository org.apache.derby.drda.NetworkServerControl start -noSecurityManager -h 0.0.0.0
Can be updated to specify higher or lower limits for the embedded Derby database server. The default is 512MB.
In general on a 4GB linux server I would allocate 2GB for CM1 1GB for Derby and leave 1GB for the operating system.
To add some more information to this, it has come to our attention that on Linux machines, the DatabaseStartup.sh script defaults to use only 128MB, as opposed to CM1’s intended default of 512MB. A fix for this is expected in CM1 2.8.
While working with Matt through an internal support ticket, Nick bumped that up to 1024MB, which, along with addressing the disk space issue in their other topic https://community.percussion.com/t/unable-to-access-percussion/192, should resolve many of Matt’s slowdown and crashing issues.
If any other customers are running CM1 on a Linux machine and are using the Derby database, they should check the DatabaseStartup.sh file at CM1’s root and verify that the line Nate referenced above is allocating at least 512MB for the database.
Nathaniel & Nate,
Thanks for bumping up that memory. We did notice fast response times. However, we’ve noticed these past few weeks that after a few days of use, Percussion will start bogging down again. To fix it, we’ve been running a quick shutdown/startup; after which, it seems to run fine again.
Jonathan mentioned in the future (if support becomes available) of possibility migrating our database from Derby to MYSQL.
In the meantime, is there anything else we can do to improve response times? It’s specifically slow when editing, saving or approving site-wide assets.
Matt,
Beyond migrating from Derby over to MySQL – which is a feature we’re targeting for the first half of 2013, as indicated in this topic: https://community.percussion.com/t/can-we-migrate-to-sql-from-derby/757 – the next obvious step would be to check your available system memory for the machine which CM1 runs off of (using the command Nate posted above), and look into the possibility of adding more system memory. I will follow-up if I think of any other ways to optimize your machine for CM1.
Nathaniel
We are experiencing the same slowdown issue with our installation of CM1. We also increased the memory allocation, but now it just takes longer for CM1 to run out of memory.
Is there a known memory leak?
Hi Conan,
When you say CM1 is running out of memory, does the UI appear to be slowing down after a while, or is it simply becoming unresponsive after a time and requiring a restart?
Can you tell me how much RAM the machine which runs CM1 has available, and if you run your CM1 database on the same machine (or if you’re using the embedded Derby database)? Additionally, you say you increased the system’s memory allocation – are you referring to the “-Xms128m -Xmx1024m” settings in the PercussionServer.bin.lax config file? If so, can you tell me what you set these to?
Thanks.
Hi Nathaniel,
Both. The UI slows down after a while, and then it becomes unresponsive and requires a restart.
Our middleware team upped the memory to 2 Gigs (not sure if Xms or Xmx) and at the moment CM1 using 2.3 Gig and 100% of the CPU time.
Hi Conan,
Without looking at your server log file to know for sure, it sounds like you are running out of Java heap space again. It is not uncommon to have to allocate 3GB to the CM1 Java process, depending on the size and quantity of the content you have in CM1 (we allocate 3GB for the CM1 instance we use to run Percusison.com, for instance).
I believe your system has 8GM of RAM available, is that correct? If so, as a next step, I would recommend upping both your Xms and your Xmx settings in that PercussionServer.bin.lax from:
“-Xms2048m -Xmx2048m”
To:
“-Xms3072m -Xmx3072m”
Be sure to stop then start CM1 after making these changes. You can find additional information about these options in this help document:
http://help.percussion.com/publishing…
If you or your middleware team have any hesitation making these changes, let me know and I can make them for you, and I’ll send you the updated file through the support ticket we already have open.
We at Embry-Riddle worldwide are having this problem again! This is getting pretty frustrating, what are our next steps?
Our middleware folks are out today so we will lose two days of development time because we cannot restart the system. Our deadlines are tight, I cannot spare two days.
Chrissy Clary,
Director, Web Services Embry-Riddle Worldwide
Claryc@erau.edu
Chrissy,
I completely understand your frustration. Do you know if your middleware people were able to make the memory allocation adjustments I recommended previously? At any rate, I’m going to reach out to you through our existing support ticket to see if there’s anything we can do to get you back in the system in the meantime.