Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[MX] How to identify the link that may be causing interrupt throttling on an MPC

0

0

Article ID: KB34609 KB Last Updated: 26 Jul 2019Version: 1.0
Summary:

When a link state change occurs or when an XFP is seen to misbehave, excessive interrupts might be generated by the physical interface (PHY) on the MPC card in MX Series routers, which results in the generation of interrupts on the MQCHIP.

This article describes the procedure to identify the mapping between the FPC, MQCHIP, and XE numbers in order to determine the potential links that are causing the throttling.

 

Symptoms:

When an MQCHIP throttle event occurs, the following log message is reported:

Jun 8 01:48:47 : %PFE-4: fpc3 MQchip 2 XE 1 Throttle: Last 1535 seconds have seen interrupt throttling at least once per second

 

Solution:

From the above log, the following can be determined:

  • FPC = 3, MQCHIP = 2 and XE = 1

To identify the mapping between the FPC, MQCHIP and XE numbers and determine the potential links that are causing the throttling, perform the following steps:

  1. Log in to the FPC3 shell and check the IFDs that are mapped to MQCHIP 2.

Since this is an MPC 3D 16x 10GE card, each MQCHIP will be mapped to four ports and MQCHIP 2 will be mapped to xe-3/2/0, xe-3/2/1, xe-3/2/2, and xe-3/2/3, as shown below:

show chassis hardware | match fpc

FPC 0            REV 16   750-038493   CACF1627          MPCE Type 2 3D Q
FPC 1            REV 16   750-038493   CACF1651          MPCE Type 2 3D Q
FPC 2            REV 08   750-063741   CAKS4200          MPCE Type 2 3D Q
FPC 3            REV 39   750-028467   CACL3287          MPC 3D 16x 10GE‚Äč   <<<<<<<< 
FPC 7            REV 20   750-024064   CABN8602          MS-DPC
FPC 9            REV 07   750-063741   CAHY2009          MPCE Type 2 3D Q

> start shell pfe network fpc3          <<< Issue this command to enter FPC3.

NPC3(JTAC000V1I vty)# show mqchip 2 ifd   <<< This command will give all the interfaces that are associated with MQCHIP 2.

  Input    IFD     IFD       LU       
  Stream  Index    Name      Sid  TClass
  ------ ------ ---------- ------ ------
    1025    191   xe-3/2/0      0     hi
    1026    191   xe-3/2/0      0    med
    1027    191   xe-3/2/0      0     lo
    1029    192   xe-3/2/1     33     hi
    1030    192   xe-3/2/1     33    med
    1031    192   xe-3/2/1     33     lo
    1033    193   xe-3/2/2     66     hi
    1034    193   xe-3/2/2     66    med
    1035    193   xe-3/2/2     66     lo
    1037    194   xe-3/2/3     99     hi
    1038    194   xe-3/2/3     99    med
    1039    194   xe-3/2/3     99     lo
    1040    191   xe-3/2/0      0   drop
    1040    192   xe-3/2/1     33   drop
    1040    193   xe-3/2/2     66   drop
    1040    194   xe-3/2/3     99   drop

  Output   IFD     IFD             Base
  Stream  Index    Name     Qsys   Qnum
  ------ ------ ---------- ------ ------
    1024    191   xe-3/2/0    MQ0      0
    1025    192   xe-3/2/1    MQ0    256
    1026    193   xe-3/2/2    MQ0    512
    1027    194   xe-3/2/3    MQ0    776

Here, interfaces xe-3/2/0, xe-3/2/1, xe-3/2/2, and xe-3/2/3 are part of MQCHIP 2.

  1. Now check the output of the following command. Each XE will be catering to two ports on the associated MQCHIP.

XE0 will cater to xe-3/2/0 and xe-3/2/1, whereas XE1 will cater to xe-3/2/2 and xe-3/2/3.

The log “fpc3 MQchip 2 XE 1 Throttle” reported XE 1 so we know that either xe-3/2/2 or xe-3/2/3 is causing the interrupts.

NPC3(JTAC000V1I vty)# show bcm8747 all

Index          fpc/pic/port   phy_addr       status
============================================================
1                3/0/0          0              OFF
2                3/0/1          1              OFF
3                3/0/2          2              OFF
4                3/0/3          3              OFF
5                3/1/0          0              ON
6                3/1/1          1              ON
7                3/1/2          2              ON
8                3/1/3          3              ON
9                3/2/0          0              ON       <<<  XE0
10               3/2/1          1              ON       <<<  XE0
11               3/2/2          2              ON       <<<  XE1
12               3/2/3          3              ON       <<<  XE1
13               3/3/0          0              ON
14               3/3/1          1              ON
15               3/3/2          2              ON
16               3/3/3          3              ON
  1. Use the below command multiple times to see if the interrupts are still increasing.

NPC3(JTAC000V1I vty)# show mqchip 2 counters interrupt

Interrupt Counters
  total intr count   : 148609048
  toe intr count     : 19814
  xe0 intr count     : 1    <<<<< Interrupt counter for XE0 
  xe1 intr count     : 148588854 <<<<< Interrupt counter for XE1 

NPC3(JTAC000V1I vty)# show mqchip 2 counters interrupt
Interrupt Counters
 total intr count   : 148638058
  toe intr count     : 19819
  xe0 intr count     : 1
  xe1 intr count     : 148617859 <<<<< Check if this is increasing with the same frequency as the logs. Then we can inspect ports xe-3/2/2 and xe-3/2/3.

From the above command, we can see that "xe1 intr count" is incrementing. So in our example, the potential candidate links that are causing the throttling are either xe-3/2/2 or xe-3/2/3. 

  1. We can narrow down the problem further by disabling xe-3/2/2 first and then disabling xe-3/2/3 next to see if the interrupts stop with either of the actions.

Note: In this case, we disabled the xe-3/2/2 interface and we can see that the interrupts stopped incrementing. So it is port xe-3/2/2 that is causing the interrupts.

labroot#set interfaces xe-3/2/2 disable

labroot# commit
re0:
configuration check succeeds
re1:
commit complete
re0:
commit complete
 
{MASTER}[edit]
labroot# run start shell pfe network fpc3
 

NPC3(JTAC000V1I vty)# show mqchip 2 counters interrupt

  Interrupt Counters
  total intr count   : 150310988
  toe intr count     : 20039
  xe0 intr count     : 1
  xe1 intr count     : 150290523
 

NPC3(JTAC000V1I vty)# show mqchip 2 counters interrupt

  Interrupt Counters
  total intr count   : 150310990
  toe intr count     : 20041
  xe0 intr count     : 1
  xe1 intr count     : 150290523  << counter stopped incrementing

  1. Now enable the interface again and perform the following steps to determine if the issue is with the transceiver or the cable.
    1. Remove the XFPs from the ports and see if the throttle messages stop.

    2. If the problem persists, re-install the XFPs back on the same port.

    3. Then remove the cable and replace it with another good cable to check if the error persists. We can also check for any problems at the remote-end XFP. If cable replacement does not stop the throttle messages, then replace the remote XFP.

    4. If the problem travels with the cable, it could be a WAN-side issue. If the problem does not travel with the cable, it could be that the XFP is faulty.

If the problem persists, contact Support to open a Technical Service Request.

 

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search