Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Junos Space] Memory and System Information Collection

0

0

Article ID: KB29084 KB Last Updated: 27 Jul 2015Version: 3.0
Summary:

This article describes the execution of a shell script used to automate data collection of process, database, and system information for Junos Space. The information collected by the script is critical for debugging and providing root-cause analysis for OutOfMemory errors in Junos Space. The script is intended to be run by JTAC engineers, who will provide the collected information to Junos Space engineering.

Symptoms:

A common result of an OutOfMemoryError condition is not being able to log into the Junos Space web user interface. Instead, the user is presented with "The username or password is incorrect." -- even for valid credentials. The error can be verified by the presence of java.lang.OutOfMemoryError: GC overhead limit exceeded entries in Space log files (for example, server.log, serviceInsight.log, and so on).

Restarting the jboss process on the affected node is usually sufficient to remedy the problem. However, important memory-state debugging information is lost due to the service restart.

Prior to rebooting any appliances or restarting jboss processes, the script should be run on all nodes in a Junos Space fabric.

Cause:

From Java documentation:

"The concurrent collector will throw an OutOfMemoryError if too much time is being spent in garbage collection: if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered, an OutOfMemoryError will be thrown. This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small. If necessary, this feature can be disabled by adding the option -XX:-UseGCOverheadLimit to the command line."

It is possible to increase the heap size as a temporary workaround. However, it is not a general solution and could cause the appliance to use swap memory, which adversely affects application performance.

Solution:

OutOfMemory errors should be considered bugs, and should be reported to engineering via a PR.

The PR should contain the archive generated by the attached shell script.

To execute the script, perform the steps below:

1) Download the script from the following URL (log in required):

https://download.juniper.net/software/space/tools/SpaceMemCollect_102.tar.gz

2) Copy SpaceMemCollect.tar.gz to each node in the Space fabric in the /home/admin/ directory.

3) SSH into the CLI of each Space node.

4) Extract the file (the archive contains SpaceMemCollect.sh):

tar zxfv SpaceMemCollect.tar.gz

5) Change the permissions and ownership of the SpaceMemCollect.sh file:

chmod 550 /home/admin/SpaceMemCollect.sh
chown root.root /home/admin/SpaceMemCollect.sh

6) Execute the shell script in the background:

/home/admin/SpaceMemCollect.sh &

7) Disown the job so that it will continue to run even if the SSH connection times out:

disown %1

The script will report progress to the CLI such as:

JTAC Junos Space Memory and System Information Collection Tool
-------------------------------------------------------------------
Files will be saved at: /var/space/...
JBoss PID is xxxxx
...
[Step 1 of xx ] Collecting jmap heap dump...
...
[Step xx of xx] Archiving the files to: /var/space/xxx.tar.gz
-------------------------------------------------------------------
Script completed

After the script has completed, you can collect the archive from the /var/space/ directory. Provide this archive to Space engineering.

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search