Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Junos] "Severe memory pressure" error when mosquitto-nossl process is run

0

0

Article ID: KB35955 KB Last Updated: 29 Jun 2020Version: 1.0
Summary:

The Message Queue Telemetry Transport (MQTT) protocol provides a lightweight method of carrying out messaging by using the publish/subscribe model. Many of the high-level languages have an MQTT library available. A Juniper Extension Toolkit (JET) application can use the corresponding library to connect to an MQTT broker that is running on Junos OS and subscribe for events.

The mosquitto-nossl process is used to back the MQTT states and is enabled by default if JET is used. This article describes the memory leak problem that is seen due to the mosquitto-nossl process, the "severe memory pressure" error that it logs, and the resolution and workaround to resolve the same.

 

Symptoms:

A leak in the mosquitto-nossl daemon will result in error messages that indicate pressure on memory utilization even when the value of low-free-pages is below the set threshold.

Log Message

Nov 27 08:15:36  rpd[14910]: RPD_KRT_Q_RETRIES: route add: Resource temporarily unavailable
Nov 27 08:15:36  rpd[14910]: RPD_KRT_Q_RETRIES: route add: Resource temporarily unavailable
Nov 27 08:15:41  kernel: rt_pfe_veto: Severe memory pressure, low-free-pages is below threshold. Op 1 err 12, rtsm_id 5:-1, msg type 2
Nov 27 08:15:41  kernel: rt_pfe_veto: free page threshold = (15519) number of pages free = (39444) number of pages on buffer cache queue = (0) curproc = rpd
Nov 27 08:15:46  kernel: rt_pfe_veto: Severe memory pressure, low-free-pages is below threshold. Op 1 err 12, rtsm_id 5:-1, msg type 2
Nov 27 08:15:46  kernel: rt_pfe_veto: free page threshold = (15519) number of pages free = (39357) number of pages on buffer cache queue = (0) curproc = rpd
Nov 27 08:15:51  kernel: rt_pfe_veto: Severe memory pressure, low-free-pages is below threshold. Op 1 err 12, rtsm_id 0:-1, msg type 2
Nov 27 08:15:51  kernel: rt_pfe_veto: free page threshold = (15519) number of pages free = (39425) number of pages on buffer cache queue = (0) curp

The above log message indicates that the swap memory resources are exhausted. There is no specific trigger for the issue.

When you check the processes list, it shows multiple process IDs for the mosquitto-nossl daemon as shown below:

user@host> show system processes extensive no-forwarding

last pid: 65143;  load averages:  0.38,  0.44,  0.42  up 241+23:42:38    13:36:54
160 processes: 2 running, 157 sleeping, 1 waiting

Mem: 1235M Active, 13G Inact, 1052M Wired, 177M Cache, 1643M Buf, 145M Free
Swap: 8192M Total, 8192M Free

  PID USERNAME     THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
   10 root           4 155 ki31     0K    64K CPU3    3    ??? 382.32% idle
13289 root           2 -26  r26   826M 44820K nanslp  3 315.8H   4.59% chassisd
13439 root           1  27    0   725M 11608K select  3 227.2H   3.17% python
   12 root           3  -8    -     0K    48K -       0   0:18   2.10% geom
64930 root           1  25    0   761M 27384K select  2   0:00   1.46% cli
15112 daemon         1  20    0  4834M  4827M select  0  66.6H   1.07% mosquitto-nossl
15114 daemon         1  20    0  4834M  4827M select  2  66.6H   1.07% mosquitto-nossl
13303 daemon         1  21    0  2426M  2414M select  3  32.2H   1.07% mosquitto-nossl

This indicates memory leak in the mosquitto-nossl daemon.

 

Solution:

The issue has been fixed in the following releases: Junos OS 16.2R2-S7, 16.2R3, 17.1R2-S7, 17.1R3, 17.2R1-S7, 17.2R2-S6, 17.2R3, 17.2X75-D110, 17.2X75-D91, 17.3R2-S3, 17.3R3, 17.4R1-S1, 17.4R2, 18.1R1, 18.2R1, and 18.2X75-D5 as documented in PR1324531.

If you continue to see the issue even on devices that are running one of the above releases, open a case with Support for further troubleshooting.

Meanwhile, if you want to temporarily prevent the daemon from re-appearing, you may select the following workaround, which prevents the mosquitto-nossl daemon from re-spawning:

  1. Log in as the root user:

user@device>start shell user root
root@host:/usr/libexec # ls -ltr | grep mos
lrwxr-xr-x  1 root  wheel    55 Mar 14 11:57 mosquitto-nossl -> /packages/mnt/junos-daemons/usr/libexec/mosquitto-nossl
lrwxr-xr-x  1 root  wheel    45 Mar 21 15:06 mosquitto -> /packages/mnt/jsd-jet-1/usr/libexec/mosquitto

root@host:/usr/libexec # mv mosquitto-nossl mosquitto-nossl-inactive
 
root@jtac-mx480-r2007:/usr/libexec # ps -aux | grep mos
daemon  19591   0.0  0.0   22736   2116  -  S     1Apr19     9:00.20 /usr/libexec/mosquitto-nossl -c /var/run/mqtt_broker_re.
daemon  20030   0.0  0.0   22736   2892  -  S     1Apr19     9:46.91 /usr/libexec/mosquitto-nossl -c /var/run/mqtt_broker_cha
daemon  20031   0.0  0.0   22736   2892  -  S     1Apr19     9:47.17 /usr/libexec/mosquitto-nossl -c /var/run/mqtt_broker_lch
root     3847   0.0  0.0   22688   2180  0  S+   12:56PM     0:00.00 grep mos

root@host:/usr/libexec # kill -9 19591
root@host:/usr/libexec # kill -9 20030
root@host:/usr/libexec # kill -9 20031
root@host:/usr/libexec # exit
 
user@host> show system processes extensive | match mos
 <Does not display any output>

 

Related Links

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search