Knowledge Search


×
 

[SBR] The web server (jetty) is using a majority of the system memory

  [KB34738] Show Article Properties


Summary:

The SBR Web Server (jetty) is seen using a large amount of memory. This article explains how to edit the Java settings to utilize less memory.

Cause:

After starting SBR, the java process will reserve a significant portion of the overall installed memory, which can leave other applications and services with limited memory utilization.

Solution:

Verify the current java Heap settings that would be edited:

Note: These outputs may be different from customer installed setups

# java -XX:+PrintFlagsFinal -version | grep -iE 'HeapSize|PermSize|ThreadStackSize'
    uintx AdaptivePermSizeWeight                    = 20               
     intx CompilerThreadStackSize                   = 0               {pd product}
    uintx ErgoHeapSizeLimit                         = 0                
    uintx HeapSizePerGCThread                       = 87241520         
    uintx InitialHeapSize                          := 260489792        
    uintx LargePageHeapSizeThreshold                = 134217728        
    uintx MaxHeapSize                              := 4169138176       
    uintx MaxPermSize                               = 174063616       {pd product}
    uintx PermSize                                  = 21757952        {pd product}
     intx ThreadStackSize                           = 1024            {pd product}
     intx VMThreadStackSize                         = 1024            {pd product}
java version "1.7.0_181"
OpenJDK Runtime Environment (rhel-2.6.14.10.el6-x86_64 u181-b00)
OpenJDK 64-Bit Server VM (build 24.181-b00, mixed mode)

top - 15:12:28 up 346 days, 23:49,  4 users,  load average: 0.00, 0.01, 0.00
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.1%sy,  0.0%ni, 99.7%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16280612k total, 15853744k used,   426868k free,   433512k buffers
Swap:  8191992k total,  1129000k used,  7062992k free, 13240068k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4129 root      20   0 6953m 224m 2420 S  0.0  1.4  90:06.47 java

# ps -ef | grep sbr
root     108719      1 99 03:43 ?        00:32:46 radius sbr.xml
root     108870      1  1 03:43 pts/2    00:00:25 /usr//bin/java -Djetty.logs=/opt/JNPRsbr/radius/website/webserver/jetty/logs (snip)

# top -b -p 108870
top - 03:58:10 up 5 days,  5:11,  4 users,  load average: 0.07, 0.09, 13.85
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.5%us,  0.4%sy,  0.0%ni, 99.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32523644k total, 17466208k used, 15057436k free,   229964k buffers
Swap: 16777212k total,     4212k used, 16773000k free, 13315612k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
108870 root      20   0 13.4g 645m  15m S  2.0  2.0   0:24.63 java

# jstat -gccapacity 108870
 NGCMN    NGCMX     NGC     S0C   S1C       EC      OGCMN      OGCMX       OGC         OC       MCMN     MCMX      MC     CCSMN    CCSMX     CCSC    YGC    FGC
169984.0 2710528.0 553984.0 20992.0 20992.0 256000.0   339968.0  5422080.0   339968.0   339968.0      0.0 1060864.0  13184.0      0.0 1048576.0   1408.0      3     0

# jmap -heap 108870
Attaching to process ID 108870, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.171-b10

using thread-local object allocation.
Parallel GC with 23 thread(s)

Heap Configuration:

   MinHeapFreeRatio         = 0
   MaxHeapFreeRatio         = 100
   MaxHeapSize              = 8327790592 (7942.0MB)
   NewSize                  = 174063616 (166.0MB)
   MaxNewSize               = 2775580672 (2647.0MB)
   OldSize                  = 348127232 (332.0MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.tools.jmap.JMap.runTool(JMap.java:201)
        at sun.tools.jmap.JMap.main(JMap.java:130)
Caused by: java.lang.RuntimeException: unknown CollectedHeap type : class sun.jvm.hotspot.gc_interface.CollectedHeap
        at sun.jvm.hotspot.tools.HeapSummary.run(HeapSummary.java:144)
        at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
        at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
        at sun.jvm.hotspot.tools.HeapSummary.main(HeapSummary.java:49)
        ... 6 more

Since JAVA Memory can be configured, the following parameters can be edited (additional information can be found in Oracle's documentation on JAVA configurations):

  • # java -X (This will show all options that can be utilized to change Heap settings)
  • # java -Xms<Minimum value, i.e. 256M> -Xmx<Maximum value, i.e. 2048M>

However, to make it easier, edit JAVA_OPTIONS:

# vi (or some other edit option) /opt/JNPRsbr/radius/website/webserver/jetty/bin/jetty.sh file

# JAVA
#   Command to invoke Java. If not set, java (from the PATH) will be used.

JAVA_OPTIONS="-Xms1g -Xmx5g -server -XX:+UseConcMarkSweepGC"
#   Extra options to pass to the JVM
#   

Be sure to save file and restart Jetty process. This can be done by restarting SBR or by executing /opt/JNPRsbr/radius/website/webserver/jetty/bin/jetty.sh restart


How did Juniper decide on the minimum value of 1GB and maximum value of 5GB?

The Java process size depends on the heap space which primarily depends on the physical memory of the machine. This can be altered by adding flags(xms and xmx) to JVM. The user can check the heap size through jstat command. NGCMX( New generation) and OGCMX(Old Generation) are ~2.5GB and ~5GB, respectively. This heap size will increase up the Java process size. Set the limit to min heap size(xms) as 1GB  and max heap size(xmx) as 4GB. After the change, NGCMX( New generation) and OGCMX(Old Generation) are reduced to ~300MB and ~4GB, respectively with java process size as ~5.5GB.

NGCMX and OGCMX are JAVA Garbage Collection Functions

NOTE: Although the properties are provided in this article, it is recommended to work with your OS Admin to ensure changes are done correctly in your environment.

Related Links: