Support Support Downloads Knowledge Base Juniper Support Portal Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Contrail] Cassandra down in Contrail analytics nodes due to java.lang.OutOfMemoryError: Java heap space



Article ID: KB35008 KB Last Updated: 11 Aug 2020Version: 5.0

An issue with Cassandra being out of memory is seen frequently in Contrail analytics nodes.

This article applies specifically to Contrail 5.X release, and provides the steps to resolve the issue while also explaining the cause.



Output of contrail status

== Contrail database ==
kafka: active
nodemgr: initializing (Cassandra state detected DOWN. )
zookeeper: active
cassandra: down



This issue is due to Cassandra being out of memory. The error messages are printed in system.log and debug.log.

ERROR [MessagingService-Incoming-/X.X.X.X] 2019-07-03 13:32:09,110 - Exception in thread Thread[MessagingService-Incoming

java.lang.OutOfMemoryError: Java heap space
        at ~[na:1.8.0_171]
        at ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.gms.GossipDigestSynSerializer.deserialize( ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.gms.GossipDigestSynSerializer.deserialize( ~[apache-cassandra-3.11.2.jar:3.11.2]
        at ~[apache-cassandra-3.11.2.jar:3.11.2]
        at ~[apache-cassandra-3.11.2.jar:3.11.2]
        at ~[apache-cassandra-3.11.2.jar:3.11.2]
        at ~[apache-cassandra-3.11.2.jar:3.11.2]



Cassandra error logs are found in debug.log and system.log. Run the following commands on analytics host to find the exact location:

find / -name debug.log
find / -name system.log

The results are similar to the example below.

Note: The container ID may differ according to the environment.

Look into the latest timestamp to understand the reason for failure:


If the error is similar to "java.lang.OutOfMemoryError: Java heap space," the steps below can be performed:

  1. Find in the host by using the command: find / -name

The output will be similar to the following:

# find / -name
  1. Edit /var/lib/docker/overlay2/<container_id>/merged/etc/cassandra/ (There are two for different containers, for example, analytics_database_cassandra or contrail_analytics_database. You may need to edit based on contrail-status output.)

  2. Edit and search for the values of MAX_HEAP_SIZE and HEAP_NEWSIZE. Set them as 16G and 2G, respectively.

Before changes:


After changes:

  1. Restart docker container by using docker restart <container name>.

This procedure needs to be performed on all Contrail analytics nodes.

Note: ​Currently MAX_HEAP_SIZE (Xmx) and NEW_HEAP_SIZE (Xms) get configured automatically during deployment.

But if users want to configure MAX_HEAP_SIZE explicitly to a higher value, then NEW_HEAP_SIZE should be configured using the following formula:

NEW_HEAP_SIZE = min (100 times the number of cores, ¼ of Heap Size)

(Reference -

For an RHOSP deployment, the following can be included in contrail-services.yaml to make it persistent for deployments:


Note: Please ensure that roles_data.yaml has matching role OS::TripleO::Services::ContrailAnalyticsDatabase for this to work.


Modification History:
  • 2019-09-12: Corrected RHOSP deployment information in the Solution.

  • 2020-10-06: section improved for clarity.

  • 2019-12-07: Added note about roles_data.yaml and note about container name.

  • 2020-08-11: Added xms formula with link


Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search