Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[SRX] High CPU utilization in Routing-Engine due to processes "swi2: netisr 0" and "irq10: em0 em1+++*"

0

0

Article ID: KB36033 KB Last Updated: 30 Jun 2020Version: 1.0
Summary:

This article explains one possible root cause for high CPU utilization in the Routing-engine (RE) of a SRX firewall when the following processes are also displaying CPU usage:

swi2: netisr 0
irq10: em0 em1+++*"
Symptoms:

Idle cpu is low and Interrupt and Kernel are high:

root@srx> show chassis routing-engine

Routing Engine status:
Slot 0:
Current state                  Master
Election priority              Master (default)
Temperature                 28 degrees C / 82 degrees F
CPU temperature             23 degrees C / 73 degrees F
DRAM                      3313 MB (16384 MB installed)
Memory utilization          16 percent
CPU utilization:
User                       5 percent
Background                 0 percent
Kernel                    30 percent   
Interrupt                 30 percent    
Idle                      35 percent    
Model                          RE-S-1800x4
Serial ID                      0000000000
Start time                     2020-05-05 04:00:56 UTC
Uptime                         8 days, 12 hours, 49 minutes, 38 seconds
Last reboot reason             0x1:power cycle/failure
Load averages:                 1 minute   5 minute  15 minute
22.42 16.35      14.46

Processes:
root@srx> show system processes extensive | except 0.0

last pid: 19437;  load averages: 22.42, 16.35, 14.46  up 8+12:50:10    16:50:36
193 processes: 3 running, 175 sleeping, 1 zombie, 14 waiting

Mem: 318M Active, 101M Inact, 159M Wired, 1016M Cache, 69M Buf, 1644M Free
Swap: 3313M Total, 3313M Free

PID USERNAME      THR PRI NICE   SIZE    RES STATE    TIME WCPU COMMAND
11 root            1 171   52     0K    12K RUN     84.7H 35.21% idle
14 root            1 -40 -159     0K    12K WAIT    29.3H 12.26% swi2: netisr 0   
20 root            1 -68 -187     0K    12K RUN     20.5H 11.28% irq11: em0 em1 em*  
2643 root            1 132    0   125M 25532K select 983:34 6.93% chassisd
18965 root            1 121    0 33840K 26748K select   0:37 2.78% mgd
12 root            1 -20 -139     0K    12K WAIT   336:52 1.95% swi7: clock sio
19394 root            1  98    0 20624K 15088K select   0:00 1.54% cli
Cause:

The swi2: netisr 0 process is a software interrupt handling thread and it is responsible for RE network traffic handling (RE-RE, RE-PFES connections, etc)​. ARP packets are sent to the RE for handling via the "swi2: netisr 0" process and a high ARP activity in the network can cause the process CPU usage to go high.

The following could be indicators of an ARP storm / L2 loop in the network:

root@srx> show system queues
[output truncated]
input protocol              bytes          max  packets max    drops
splfwdq                         0      1000000        0 1000        0
splnetq                         0      1000000        0 1000        0
arpintrq                        0         3000        0 50     5419402


ARP activity can be monitored with the following command:

root@srx> show system statistics arp
arp:
[output truncated]
482411627 datagrams received
4978470 ARP requests received
316177 ARP replies received
29134 resolution request received
402768 received proxy requests
7509132 datagrams with my own hardware address
34289 datagrams for an address not on the interface
1972855 datagrams with source address duplicate to mine
467197936 datagrams which were not for me
359441 packets discarded waiting for resolution
39 packets sent after waiting for resolution
3733785 ARP requests sent
5381238 ARP replies sent
Solution:
1. Confirm the active interfaces in the device:
root@srx> show interfaces extensive | match "Physical interface" | match "Physical link is Up"
      Physical interface: reth1  , Enabled, Physical link is Up 
      Physical interface: reth3  , Enabled, Physical link is Up
      Physical interface: reth7  , Enabled, Physical link is Up

2. Run "monitor traffic interface" on these interfaces for 10 seconds and confirm the total number of captured packets. Note this command only capture packets source or destined to the RE.
root@srx> show interfaces terse | match reth1
reth1.0              up    up   inet     10.10.10.1/24

root@srx> monitor traffic interface reth1 no-resolve

[output truncated]
18:34:53.741134  In arp who-has 10.10.10.43 tell 10.10.10.1
18:34:53.741181  In arp who-has 10.10.10.163 tell 10.10.10.1
18:34:53.741193  In arp who-has 10.10.10.195 tell 10.10.10.1
18:34:53.742219  In arp who-has 10.10.10.54 tell 10.10.10.1
18:34:53.742237  In arp who-has 10.10.10.51 tell 10.10.10.1
18:34:53.842099  In arp who-has 10.10.10.144 tell 10.10.10.1
18:34:53.842109  In arp who-has 10.10.10.178 tell 10.10.10.1
18:34:53.843184  In arp who-has 10.10.10.175 tell 10.10.10.1​
[output truncated]
20955 packets received by filter
0 packets dropped by kernel

A high amount of packets, captured in 10 seconds, can be seen above and these have to be processed by the RE. Also ingress ARP requests with the source address of reth1.0 are noticed, meaning that these packets were sent from the SRX and looped back to the device. The L2 network has to be investigated because there is a network loop.

Related Links

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search