Inter-member traffic flow can break on a QFX5110-VC, and QFX5110-VCF, when doing one of the following actions:
- Rebooting the VC or VCF
- Removing and reinserting the VCP cables
PROBLEM:
On issuing a reboot on a QFX5110-VC or a QFX5110-VCF, or when removing and then reinserting the VCP cables, SDK gets into an incorrect
'modport' setting.
This breaks traffic flow between certain members of the QFX5110-VC or QFX5110-VCF. Once either the QFX5110-VC or QFX5110-VCF has re-converged after the reboot, or after reinserting the cables, the virtual chassis VCP (virtual chassis ports) ports stop forwarding traffic received on the local member to a different member.
- This issue might affect traffic between certain VC/VCF members and not between all members.
- It does not affect traffic traversing a standalone QFX5110, or traffic ingressing, and egressing the VC/VCF via the same member.
SOLUTION:
PR1350716 - The issue is fixed for both the VC and VCF starting in JunOS 17.3R2-S2, 17.3R3 and 17.4R2 onwards in the respective release branches.
- QFX5110-VC and QFX5110-VCF Fixed in JunOS Software 17.3R2-S2, 17.3R3, 17.4R2
WORKAROUND:
- The workaround to resume inter-member traffic flow is to delete and re-configure the VC ports on the VC/VCF members between which traffic forwarding is broken.
PROCEDURE:
1. Delete and re-configure the VC port on either of the affected members like member 0 here:
root@dc-core-1> show virtual-chassis vc-port
fpc0:
--------------------------------------------------------------------------
Interface Type Trunk Status Speed Neighbor
or ID (mbps) ID Interface
PIC / Port
0/1 Configured -1 Up 40000 1 vcp-255/0/48
0/2 Configured -1 Up 40000 2 vcp-255/0/2
root@dc-core-1> request virtual-chassis vc-port delete pic-slot 0 port 1 member 0
fpc0:
--------------------------------------------------------------------------
Port deletion initiated, use cmd show virtual-chassis vc-port to verify
root@dc-core-1> request virtual-chassis vc-port set pic-slot 0 port 1 member 0
fpc0:
--------------------------------------------------------------------------
Port conversion initiated, use show virtual-chassis vc-port to verify
Interface: xe-1/0/46, Enabled, Link is Up
Encapsulation: Ethernet, Speed: 10000mbps
Traffic statistics: Current delta
Input bytes: 157167044416 (761999176 bps) [413185728]
Output bytes: 19848576069 (762000752 bps) [373571840]
Input packets: 2455735062 (1488283 pps) [6480497]
Output packets: 310133751 (1488282 pps) [5837059] >>>>>>>>>Traffic flow recovered
Error statistics:
Input errors: 0 [0]
Input drops: 0 [0]
Input framing errors: 0 [0]
Policed discards: 0 [0]
L3 incompletes: 0 [0]
L2 channel errors: 0 [0]
L2 mismatch timeouts: 0 Carrier transitions: [0]
OBSERVATIONS:
root@dc-core-1> show virtual-chassis
Preprovisioned Virtual Chassis
Virtual Chassis ID: 8be6.8763.4c4c
Virtual Chassis Mode: Enabled
Mstr Mixed Route Neighbor List
Member ID Status Serial No Model prio Role Mode Mode ID Interface
0 (FPC 0) Prsnt WT3717060014 qfx5110-32q 129 Backup N VC 1 vcp-255/0/1
2 vcp-255/0/2
1 (FPC 1) Prsnt WS3717190092 qfx5110-48s-4c 0 Linecard N VC 0 vcp-255/0/48
2 vcp-255/0/50
2 (FPC 2) Prsnt WT3717190092 qfx5110-32q 129 Master* N VC 1 vcp-255/0/1
0 vcp-255/0/2
+---------------------------+xe-0/0/5:0
| QFX5110VC |
| FPC0 +----------------------> jtac-Traffic_Generator-SPT-N11U-r001(3/1)
| |
+---------------------------+
| |xe-1/0/46
| FPC1 +---------------------->jtac-Traffic_Generator-SPT-N11U-r001(1/1)
+---------------------------+
| |xe-2/0/5:0
| FPC2 +---------------------->jtac-Traffic_Generator-N11U-r003(5/3)
+---------------------------+
The following traffic streams have been set up for testing:
1. Traffic from member1 to member2: jtac-Traffic_Generator-SPT-N11U-r001 (1/1)--->jtac-Traffic_Generator-SPT-N11U-r003 (5/3)
2. Traffic from member0 to member1: jtac-Traffic_Generator-SPT-N11U-r001 (3/1)--->jtac-Traffic_Generator-SPT-N11U-r001 (1/1)
3. Traffic from member2 to member0: jtac-Traffic_Generator-SPT-N11U-r003 (5/3)--->jtac-Traffic_Generator-SPT-N11U-r001 (3/1)
TRIGGER:
Reboot the QFX5110-VC.
root@dc-core-1> request system reboot all-members
IMPACT OBSERVED:
1. In this case, we observed that traffic from member 0 to member 1 and traffic between 2 and 0 would be broken.
2. Traffic between Member 1 to Member 2 is not impacted.
The following is a snippet of the impact on traffic from member 0 to member 1.
Interface: xe-0/0/5:0, Enabled, Link is Up
Encapsulation: Ethernet, Speed: 10000mbps
Traffic statistics: Current delta
Input bytes: 122516383744 (762121024 bps) [41368660864]
Output bytes: 79779942 (304 bps) [19680]
Input packets: 1914318487 (1488516 pps) [646409604]>>>>> Traffic to be forwarded to member1
Output packets: 1246372 (0 pps) [243]
Error statistics:
Input errors: 0 [0]
Input drops: 0 [0]
Input framing errors: 0 [0]
Policed discards: 0 [0]
L3 incompletes: 0 [0]
L2 channel errors: 0 [0]
L2 mismatch timeouts: 0 Carrier transitions: [0]
root@fpc0>monitor interface traffic
Interface Link Input packets (pps) Output packets (pps)
vcp-255/0/1 Up 985 (0) 9937819 (0) >>>>> No traffic is being forwarded out by
vcp-255/0/2 Up 24931717 (36) 9961997 (0) vcp-255/0/1 to member1
gr-0/0/0 Up 0 (0) 0 (0)
et-3/0/0 Down 0 (0) 0 (0)
pfh-3/0/0 Up 0 0
xe-3/0/5:0 Up 0 (0) 0 (0)
xe-3/0/5:1 Down 0 (0) 0 (0)
xe-3/0/5:2 Down 0 (0) 0 (0)
xe-3/0/5:3 Down
Interface: xe-1/0/46, Enabled, Link is Up
Encapsulation: Ethernet, Speed: 10000mbps
Traffic statistics: Current delta
Input bytes: 77574200704 (761846160 bps) [0]
Output bytes: 34939942 (304 bps) [0]
Input packets: 1212072513 (1487981 pps) [0]
Output packets: 545816 (0 pps) [0] >>>> No traffic forwarded out of this port
Error statistics:
Input errors: 0 [0]
Input drops: 0 [0]
Input framing errors: 0 [0]
Policed discards: 0 [0]
L3 incompletes: 0 [0]
L2 channel errors: 0 [0]
L2 mismatch timeouts: 0 Carrier transitions: [0]