6.7 out of memory errors

Rhythmyx Technical Discussion
1

We have a system just been upgraded from 6.5.2 to 6.7 and have the following error in the log file, which produces similar out of memory errors when subsequently a scheduled publish attempts to run:

2009-12-15 04:36:43,559 INFO [STDOUT] Exception in thread “ContainerBackgroundProcessor[StandardEngine[jboss.web]]”
2009-12-15 04:36:43,559 INFO [STDOUT] java.lang.OutOfMemoryError: GC overhead limit exceeded
2009-12-15 04:37:18,715 INFO [STDOUT] Exception in thread “Timer-2”
2009-12-15 04:37:18,715 INFO [STDOUT] at org.apache.catalina.core.ContainerBase.findChildren(ContainerBase.java:828)
2009-12-15 04:37:18,715 INFO [STDOUT] at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1574)
2009-12-15 04:37:18,715 INFO [STDOUT] at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577)
2009-12-15 04:37:18,715 INFO [STDOUT] at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577)
2009-12-15 04:37:31,715 INFO [STDOUT] at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1557)
2009-12-15 04:37:31,715 INFO [STDOUT] at java.lang.Thread.run(Unknown Source)
2009-12-15 04:37:31,715 INFO [STDOUT] java.lang.OutOfMemoryError: GC overhead limit exceeded
2009-12-15 04:38:37,075 INFO [org.springframework.jms.listener.DefaultMessageListenerContainer] Setup of JMS message listener invoker failed - trying to recover: org.jboss.mq.SpyJMSException: Cannot receive ; - nested throwable: (java.lang.OutOfMemoryError: GC overhead limit exceeded)
2009-12-15 04:39:04,121 INFO [org.springframework.jms.listener.DefaultMessageListenerContainer] Setup of JMS message listener invoker failed - trying to recover: org.jboss.mq.SpyJMSException: Cannot receive ; - nested throwable: (java.lang.OutOfMemoryError: GC overhead limit exceeded)
2009-12-15 04:39:16,746 INFO [STDOUT] Exception in thread “sys_emailMessageHandlerContainerQ-3”
2009-12-15 04:39:16,746 INFO [STDOUT] java.lang.OutOfMemoryError: GC overhead limit exceeded
2009-12-15 04:39:41,543 ERROR [org.quartz.core.ErrorLogger] An error occured while scanning for the next trigger to fire.
org.quartz.JobPersistenceException: Failed to obtain DB connection from data source ‘rhythmyxDataSource’: java.lang.OutOfMemoryError: GC overhead limit exceeded [See nested exception: java.lang.OutOfMemoryError: GC overhead limit exceeded]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.getConnection(JobStoreSupport.java:640)
at org.quartz.impl.jdbcjobstore.JobStoreTX.getNonManagedTXConnection(JobStoreTX.java:72)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3659)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2654)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:235)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

Does this provide any clues as to what my problem might be?

Andrew

2

See http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html#par_gc.oom

Although that article provides info on how to disable the feature, I think it’d be better if you looked at tuning the overall memory management of your JVM.

3

Did you have a <rxroot>/RhythmyxServer.ja file with memory parameters in it in the 6.52 version of your environment?

4

Yes, probably but I think it was also standard.

We’ve changed this line in RhythmyxServer.lax to this, which appears to be working well so far:

lax.nl.java.option.additional=-Dprogram.name=RhythmyxServer.exe -Xms128m -Xmx1024m -Djava.endorsed.dirs=“AppServer/lib/endorsed” -Djava.library.path=./bin -server -XX:PermSize=32m -XX:MaxPermSize=512m -enableassertions -Xms128m -Xmx1024m -Xss2m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+ExplicitGCInvokesConcurrent

I don’t know why the -Xms and -Xmx are specified twice in the standard file but kept it consistent.

5

Try modifying your RhythmyxServer.lax to reflect the following then restart:

lax.nl.java.option.additional=-Dprogram.name=RhythmyxServer.exe -Xms128m -Xmx1024m -Djava.endorsed.dirs=“AppServer/lib/endorsed” -Djava.library.path=./bin -server -XX:PermSize=32m -XX:MaxPermSize=128m -enableassertions -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+ExplicitGCInvokesConcurrent

If you’re still having problems after restarting, then you should submit a ticket with TS so we can troubleshoot further.