When a link state change occurs or when an XFP is seen to misbehave, excessive interrupts might be generated by the physical interface (PHY) on the MPC card in MX Series routers, which results in the generation of interrupts on the MQCHIP.
This article describes the procedure to identify the mapping between the FPC, MQCHIP, and XE numbers in order to determine the potential links that are causing the throttling.
When an MQCHIP throttle event occurs, the following log message is reported:
Jun 8 01:48:47 : %PFE-4: fpc3 MQchip 2 XE 1 Throttle: Last 1535 seconds have seen interrupt throttling at least once per second
From the above log, the following can be determined:
To identify the mapping between the FPC, MQCHIP and XE numbers and determine the potential links that are causing the throttling, perform the following steps:
- Log in to the FPC3 shell and check the IFDs that are mapped to MQCHIP 2.
Since this is an MPC 3D 16x 10GE card, each MQCHIP will be mapped to four ports and MQCHIP 2 will be mapped to xe-3/2/0, xe-3/2/1, xe-3/2/2, and xe-3/2/3, as shown below:
show chassis hardware | match fpc
FPC 0 REV 16 750-038493 CACF1627 MPCE Type 2 3D Q
FPC 1 REV 16 750-038493 CACF1651 MPCE Type 2 3D Q
FPC 2 REV 08 750-063741 CAKS4200 MPCE Type 2 3D Q
FPC 3 REV 39 750-028467 CACL3287 MPC 3D 16x 10GE <<<<<<<<
FPC 7 REV 20 750-024064 CABN8602 MS-DPC
FPC 9 REV 07 750-063741 CAHY2009 MPCE Type 2 3D Q
> start shell pfe network fpc3 <<< Issue this command to enter FPC3.
NPC3(JTAC000V1I vty)# show mqchip 2 ifd <<< This command will give all the interfaces that are associated with MQCHIP 2.
Input IFD IFD LU
Stream Index Name Sid TClass
------ ------ ---------- ------ ------
1025 191 xe-3/2/0 0 hi
1026 191 xe-3/2/0 0 med
1027 191 xe-3/2/0 0 lo
1029 192 xe-3/2/1 33 hi
1030 192 xe-3/2/1 33 med
1031 192 xe-3/2/1 33 lo
1033 193 xe-3/2/2 66 hi
1034 193 xe-3/2/2 66 med
1035 193 xe-3/2/2 66 lo
1037 194 xe-3/2/3 99 hi
1038 194 xe-3/2/3 99 med
1039 194 xe-3/2/3 99 lo
1040 191 xe-3/2/0 0 drop
1040 192 xe-3/2/1 33 drop
1040 193 xe-3/2/2 66 drop
1040 194 xe-3/2/3 99 drop
Output IFD IFD Base
Stream Index Name Qsys Qnum
------ ------ ---------- ------ ------
1024 191 xe-3/2/0 MQ0 0
1025 192 xe-3/2/1 MQ0 256
1026 193 xe-3/2/2 MQ0 512
1027 194 xe-3/2/3 MQ0 776
Here, interfaces xe-3/2/0, xe-3/2/1, xe-3/2/2, and xe-3/2/3 are part of MQCHIP 2.
- Now check the output of the following command. Each XE will be catering to two ports on the associated MQCHIP.
XE0 will cater to xe-3/2/0 and xe-3/2/1, whereas XE1 will cater to xe-3/2/2 and xe-3/2/3.
The log “fpc3 MQchip 2 XE 1 Throttle” reported XE 1 so we know that either xe-3/2/2 or xe-3/2/3 is causing the interrupts.
NPC3(JTAC000V1I vty)# show bcm8747 all
Index fpc/pic/port phy_addr status
============================================================
1 3/0/0 0 OFF
2 3/0/1 1 OFF
3 3/0/2 2 OFF
4 3/0/3 3 OFF
5 3/1/0 0 ON
6 3/1/1 1 ON
7 3/1/2 2 ON
8 3/1/3 3 ON
9 3/2/0 0 ON <<< XE0
10 3/2/1 1 ON <<< XE0
11 3/2/2 2 ON <<< XE1
12 3/2/3 3 ON <<< XE1
13 3/3/0 0 ON
14 3/3/1 1 ON
15 3/3/2 2 ON
16 3/3/3 3 ON
-
Use the below command multiple times to see if the interrupts are still increasing.
NPC3(JTAC000V1I vty)# show mqchip 2 counters interrupt
Interrupt Counters
total intr count : 148609048
toe intr count : 19814
xe0 intr count : 1 <<<<< Interrupt counter for XE0
xe1 intr count : 148588854 <<<<< Interrupt counter for XE1
NPC3(JTAC000V1I vty)# show mqchip 2 counters interrupt
Interrupt Counters
total intr count : 148638058
toe intr count : 19819
xe0 intr count : 1
xe1 intr count : 148617859 <<<<< Check if this is increasing with the same frequency as the logs. Then we can inspect ports xe-3/2/2 and xe-3/2/3.
From the above command, we can see that "xe1 intr count" is incrementing. So in our example, the potential candidate links that are causing the throttling are either xe-3/2/2 or xe-3/2/3.
- We can narrow down the problem further by disabling xe-3/2/2 first and then disabling xe-3/2/3 next to see if the interrupts stop with either of the actions.
Note: In this case, we disabled the xe-3/2/2 interface and we can see that the interrupts stopped incrementing. So it is port xe-3/2/2 that is causing the interrupts.
labroot#set interfaces xe-3/2/2 disable
labroot# commit
re0:
configuration check succeeds
re1:
commit complete
re0:
commit complete
{MASTER}[edit]
labroot# run start shell pfe network fpc3
NPC3(JTAC000V1I vty)# show mqchip 2 counters interrupt
Interrupt Counters
total intr count : 150310988
toe intr count : 20039
xe0 intr count : 1
xe1 intr count : 150290523
NPC3(JTAC000V1I vty)# show mqchip 2 counters interrupt
Interrupt Counters
total intr count : 150310990
toe intr count : 20041
xe0 intr count : 1
xe1 intr count : 150290523 << counter stopped incrementing
- Now enable the interface again and perform the following steps to determine if the issue is with the transceiver or the cable.
-
Remove the XFPs from the ports and see if the throttle messages stop.
-
If the problem persists, re-install the XFPs back on the same port.
-
Then remove the cable and replace it with another good cable to check if the error persists. We can also check for any problems at the remote-end XFP. If cable replacement does not stop the throttle messages, then replace the remote XFP.
-
If the problem travels with the cable, it could be a WAN-side issue. If the problem does not travel with the cable, it could be that the XFP is faulty.
If the problem persists, contact Support to open a Technical Service Request.