Deteriorating performance as you work on templates

Rhythmyx Technical Discussion
1

Hello,

Does anyone else experience deteriorating performance as they work on templates? On our Rhythmyx 6.5.2 system, using an Oracle 11g RAC for the database, it can quickly become unusable if making lots of small changes, for example when tinkering with HTML/CSS/JavaScript. Experimenting this morning, I experienced the following:
[ul]
[li]After restarting the Rhythmyx server, a page takes around 1.5 seconds to assemble and download to my web browser after refreshing the default full page preview (just the HTML, not including images.)
[/li][li]After making any type of change (even just adding a single character) to the page template in Workbench, it takes between 4 and 5 seconds the first time the preview is refreshed.
[/li][li]Subsequent refreshes return to taking around 1.5 seconds.
[/li][li]This pattern continues as I make small changes, save, refresh the preview until…
[/li][li]After about 20 cycles of modifying, saving then refreshing the preview, it starts to take longer. The first refresh after saving a modification to the template takes around 20 seconds. Subsequent refreshes take from 5 to 10 seconds. And it takes longer for Workbench to save the template each time.
[/li][li]Eventually it gets so slow that it fails to assemble at all. The worst example was a few days ago, when I gave up and went home. The next day, the refresh I had started before leaving had taken 9.2 minutes before failing. In the console.log the following was recorded:
[/li]

16:34:46,928 ERROR [PSAssemblerBase] Problem while assembling task: java.util.concurrent.FutureTask@127c440
java.lang.OutOfMemoryError: Java heap space
16:34:47,038 ERROR [PSAssemblyServlet] Problem assembling item (1733): Index: 0, Size: 0

[/ul]
Checking the amount of available physical memory on the (Windows) machine on which Rhythmyx is running reveals that the above usage takes up about 120 MB of the available physical memory and 36 MB of the available virtual memory (obviously other processing are running, but I was the only user on this, our development system, which has a total of 750 MB of physical memory - more than the recommended 512 MB quoted in the manuals.)

Does anyone have any suggestions or similar experiences?

Thanks,

Andrew.

2

Hi Andrew,

This issue is going to be researched by Support. We will follow up directly with you shortly…please prepare a copy of your Rx Tree and DB back to ftp to us.

Regards
Daniel

3

We’ve been experiencing a similar problem, and the only way I’ve managed to solve it is to have the server rebooted. I’ve a hunch that the memory allocation for the Java Runtime Environment may not be as large as it could be on our server, which may be a factor.

Does Workbench have any known memory leaks?

We’re running on Red Hat Enterprise Linux 5.1, Oracle 10.2 10GR2

4

I think the default java heap size may be 128 megs.

Try adding the following to your <RhythmyxRoot>/RhythmyxServer.ja

-Xmx512m

and restart your server.

We’re at 1024 megs max on our development and 2048 on our production

5

Remember also the editing templates (in the workbench) also eats up Permgen space (as well as heap space).

There are options in the JA file for increasing permgen space as well as heap space, and I recommend that you try that if increasing the heap space doesn’t resolve it.

Dave

6

That sounds promising. However, we don’t seem to have a RhythmyxServer.ja
file in the rhythmyx root. Is this where it should be? Is this a file that isn’t used by default but needs to be created? Is there an example file that I could show to our IT department?

cheers
Matt

7

RhythmyxServer.ja is mentioned in the “Getting Started with Rhythmyx,” page 108 for 6.5 and 6.5.2, page 98 for 6.6. Looks like there are different instructions for Windows vs. Unix, so my earlier instructions may not directly apply.

8

Rhythmyxserver.ja is just a text file that sits in the root of your Rhythmyx directory, below is an example of the content contained within;

%IF_EXISTS%(“INIT_JAVA_HEAP”, “@INIT_JAVA_HEAP@512m”)
%IF_EXISTS%(“MAX_JAVA_HEAP”, “@MAX_JAVA_HEAP@1536m”)

Please make sure that you do not set your MAX_JAVA_HEAP to more than 70% of the total allowable RAM on the server.

If you set the MAX_JAVA_HEAP too high you may experience the Rhythmyx Server console opening and immediately closing…if so, back off on the Max number it should start up…

Any problems with this please submit them via the normal support channels and we’ll open a new TAR.

9

We are seeing the exact symptoms in our production environment.

Rhythymx: 6.5.2
OS: Enterprise Linux Enterprise Linux AS release 4 (October Update 6)
Database: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0

Java Heap Error (I was using the workbench in the same manner as described by Andrew):

2009-03-30 21:21:18,315 ERROR [PSVelocityAssembler] Problem assembling output for item: 2-101-2018 with template: worldwide
java.lang.OutOfMemoryError: Java heap space
2009-03-30 21:22:07,704 ERROR [PSAssemblyServlet] Problem assembling item (2018): Problem assembling output for item: 2-101-2018 with template: worldwide exception: Java heap space see log for stack trace
2009-03-30 21:27:38,388 INFO  [PSPublisherService] Getting content list data took <Stopwatch stopped elapsed 9,991.8 ms> for 0 items

RhythymxServer.js content before updating:

-Djava.io.tmpdir=/cms/testdir1

Ensuring that an increased heap size does not exceed 70%:

-bash-3.00$ /cms/Rhythmyx/JRE/bin/java -mx4024m
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
-bash-3.00$ /cms/Rhythmyx/JRE/bin/java -mx1024m

Available Memory:

top - 10:38:20 up 278 days, 8 min,  4 users,  load average: 0.38, 0.18, 0.13
Tasks: 128 total,   1 running, 127 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0% us,  0.0% sy,  0.0% ni, 100.0% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:   4147744k total,  3373640k used,   774104k free,   181604k buffers
Swap: 16777208k total,      232k used, 16776976k free,  1546876k cached

RhythymxServer.js content after updating:

-Djava.io.tmpdir=/cms/testdir1 -Xmx1024m

After implementing the change in the Test and Production systems and restarting I am not seeing a change in the Rhythmyx process:

username   18039 79.1 11.1 1542208 460800 pts/4 Sl  11:10   2:08 /cms/Rhythmyx/JRE/bin/java -cp :JRE/lib/tools.jar:AppServer/bin/run.jar: -Dprogram.name=RhythmyxServer.exe -Xms128m -Xmx512m -Djava.endorsed.dirs=AppServer/lib/endorsed -Dja

Did anyone notice a difference after updating the RhythmyxServer.js file?

10

Quick follow-up on the documentation I quoted in my previous post…

Based on a ticket I have open with Percusion TS, Percussion CM 6.6 no longer uses RhthmyxRoot/RhythmyxServer.ja. Instead, it uses RhythmyxRoot/RhythmyxServer.bin.lax. Look under

#   LAX.NL.JAVA.OPTION.ADDITIONAL
#   -----------------------------
#   Java arguments
11

[QUOTE=DBaggs;6924]Rhythmyxserver.ja is just a text file that sits in the root of your Rhythmyx directory, below is an example of the content contained within;

%IF_EXISTS%(“INIT_JAVA_HEAP”, “@INIT_JAVA_HEAP@512m”)
%IF_EXISTS%(“MAX_JAVA_HEAP”, “@MAX_JAVA_HEAP@1536m”)

[/QUOTE]

Thanks for this. It looks quite different from how I imagined. I’ve asked our IT Services to create a RhythmyxServer.ja file with the following content:

-Xms512M
-Xmx512M

This didn’t seem to make much difference, so I’ve now asked them to increase it to 1024MB (our server has 16GB of RAM). Is the syntax above better? Is it applicable to 6.5.2?

Cheers
Matt

12

You can only go up to about 1.5 GB. It’s a java limitation regardless of how much memory you have on the box. In the RhythmyxServer.ja I would use:

-Xmx1536m

instead of:

%IF_EXISTS%(“INIT_JAVA_HEAP”, “@INIT_JAVA_HEAP@512m”)
%IF_EXISTS%(“MAX_JAVA_HEAP”, “@MAX_JAVA_HEAP@1536m”)

13

Thanks for that. I’ll pass this on to our IT guys.

cheers
Matt

14

Was there ever any resolution on this thread or another similar one? While it seems that throwing more space at the process is great, it doesn’t address the underlying problem that the performance does deteriorate when making changes to Templates and/or Content Types… Is there a memory leak somewhere in the app that is addressed between versions 6.5 and 6.7, or does 6.7 also exhibit this issue?

In my 6.5 installations, I’ve had to resort to only making changes on backup servers, then MSMing them to the main one. If I do make changes on the main server, I have to restart the java process after about 50 changes in order to fix performance issues.

15

This issue was addressed in Rx 6.52 through a series of patches earlier in the year. Please contact TS to receive access to the latest patch.

On another note, creating or modify objects on PROD without testing in another environment first can be very dangerous and is not recommended. Recommended procedure for creating/modifying server objects is to do whatever work you need on a DEV environment, test, migrate the new object to a TEST environment (clone of PROD) and test to make sure things transfer and work as expected, and then transfer the fully tested and complete objects to PROD using MSM.