Knowledge Search


×
 

PR1447170 - KRT queue built-up due to resource exhaustion when use "protocol mpls sensor-based-stats" and "ldp-tunneling"

  [TSB17604] Show Article Properties


Alert Type:
PSN - Product Support Notification
Product Affected:
All JUNOS platforms using "protocol mpls sensor-based-stat" and "ldp-tunneling"
Alert Description:
When using "set protocol mpls sensor-based-stats" and "ldp-tunneling", every time an LSP is added or changed, part of its data structure is not freed which will cause the resources to be exhausted. The relevant data structure is called "Generic config" or "gencfg". Junos software imposes a maximum limit of 2000000 entries for the gencfg data structure. Once the system reaches the limit, the system can no longer add a new LSP to the PFE causing transit packets being dropped.
The following are syslog entries indicating that the routing engine is running out of resources

<syslog>
Jul 1 11:32:00 router1 rpd[2595]: %DAEMON-3-RPD_KRT_Q_RETRIES: Stats Id Group ADD: Cannot allocate memory
Jul 1 11:32:00 router1 /kernel: %KERN-3: gencfg_peer_update Cannot retrieve gencfg node from ifstate
Jul 1 11:32:00 router1 last message repeated 3931 times
Jul 1 11:32:00 router1 agentd[2636]: %DAEMON-3: failed to add sensor 281475513713550
</syslog>

To check if the number of "Generic config" entries in used, you can use the following command "ifsmon -Id" - these are a capital "i" and a lowercase "d" in the argument of "ifsmon". In this example, the number of "Generic config" entries is at the maximum and no more entry can be allocated.

cli> start shell user root
# ifsmon -Id | grep "Generic config"

Generic config : 2000027 1‚Äč

Rate of accumulation of the Generic config entries can be determined by collecting the information multiple times. During a router's normal operation, the number of "Generic config" counters can vary depending on the number of LSPs, LSP activities, and how often configuration changed. However, the number of entries should not be trending upward.

In a large network, it can take approximately 30 weeks for the "Generic config" entries to reach its limit.

Solution:
If you are using "set protocol mpls sensor-based-stats" and "ldp-tunneling" configuration, you need to monitor the "Generic config" resource.

If there is a tendency for the "Generic config" to accumulate, we recommend that you take corrective actions as follow:
1. Determine the amount of "Generic config" being used by the system and the rate of accumulation
2. If the number of "Generic config" is more than 1000000 (one million entries) and the rate of accumulation is positive
Disable the "sensor-based-stats" configuration to stop the accumulation of "Generic config" entries and plan for a maintenance window to restart RPD to reclaim the entries.
Implementation:
This issue is being tracked via PR1447170. For latest update and availability of the fix, please consult PR1447170 at https://prsearch.juniper.net/PR1447170
Modification History:
First publication date 2019-08-02
Related Links: