This article explains why a mismatched state may be seen for LACP bundles on QFX5K devices by demonstrating the working of the core isolation feature with LACP bundles in a setup with EVPN configuration. It also suggests a couple of workarounds that could be used to prevent the mismatched LACP bundle state.
Note: The problem was observed in Junos OS release 17.3R3-S2.
Consider the following topology:
Spines 01 spine 02
| |
| |
| |
Leaf 01 ---------------|
ae |
|
|
Host
In the above topology, Leaf 01 on the QFX5100 device is connected to two spines. When the leaf loses connectivity to both spines, the EVPN core isolation feature is triggered and the LACP bundles (ae) are brought down (moving the Mux state to “Waiting”) so that no traffic enters Leaf 01 (basically isolating the QFX5100 device).
Note: By default, the core isolation feature is enabled on QFX5100 devices.
However, if the LACP configuration in the leaf has the minimum-links configuration enabled, the leaf will keep the ae bundle up and in the "collecting and distributing" state, a behavior that indicates a mismatched LACP bundle state. For on the peer device (works fine) as shown below, the state is “Attached,” which means no traffic is entering the leaf.
QFX5100 device showing collecting and distributing state
show lacp interfaces
Feb 11 17:36:50
Aggregated interface: ae1
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
xe-0/0/4 CDN Actor No No Yes Yes No Yes Fast Active
xe-0/0/4 CDN Partner No No No No Yes Yes Fast Active <<<<<<<<<<<<<<<
xe-0/0/5 CDN Actor No No Yes Yes No Yes Fast Active
xe-0/0/5 CDN Partner No No No No Yes Yes Fast Active
xe-0/0/6 CDN Actor No No Yes Yes No Yes Fast Active
xe-0/0/6 CDN Partner No No No No Yes Yes Fast Active
LACP protocol: Receive State Transmit State Mux State
xe-0/0/4 Current Fast periodic Collecting distributing
xe-0/0/5 Current Fast periodic Collecting distributing
xe-0/0/6 Current Fast periodic Collecting distributing
Aggregated interface: ae2
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
xe-0/0/17 CDN Actor No No Yes Yes No Yes Fast Active
xe-0/0/17 CDN Partner No No No No Yes Yes Fast Active
xe-0/0/18 CDN Actor No No Yes Yes No Yes Fast Active
xe-0/0/18 CDN Partner No No No No Yes Yes Fast Active
xe-0/0/19 CDN Actor No No Yes Yes No Yes Fast Active
xe-0/0/19 CDN Partner No No No No Yes Yes Fast Active
LACP protocol: Receive State Transmit State Mux State
xe-0/0/17 Current Fast periodic Collecting distributing
xe-0/0/18 Current Fast periodic Collecting distributing
xe-0/0/19 Current Fast periodic Collecting distributing
LACP peer showing attached state to indicate no traffic entering the leaf
show lacp interfaces ae2
Aggregated interface: ae2
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
xe-0/0/6 Actor No No No No Yes Yes Fast Active
xe-0/0/6 Partner No No Yes Yes No Yes Fast Active
xe-0/0/7 Actor No No No No Yes Yes Fast Active
xe-0/0/7 Partner No No Yes Yes No Yes Fast Active
xe-0/0/8 Actor No No No No Yes Yes Fast Active
xe-0/0/8 Partner No No Yes Yes No Yes Fast Active
xe-0/0/9 Actor No No No No Yes Yes Fast Active
xe-0/0/9 Partner No No Yes Yes No Yes Fast Active
xe-0/0/10 Actor No No No No Yes Yes Fast Active
xe-0/0/10 Partner No No Yes Yes No Yes Fast Active
xe-0/0/11 Actor No No No No Yes Yes Fast Active
xe-0/0/11 Partner No No Yes Yes No Yes Fast Active
LACP protocol: Receive State Transmit State Mux State
xe-0/0/6 Current Fast periodic Attached
xe-0/0/7 Current Fast periodic Attached
xe-0/0/8 Current Fast periodic Attached
ae bundle configuration on both devices with minimum-links enabled
QFX
set interfaces ae1 esi 00:00:00:00:00:00:00:50:33:01
set interfaces ae1 esi all-active
set interfaces ae1 aggregated-ether-options minimum-links 2
set interfaces ae1 aggregated-ether-options lacp active
set interfaces ae1 aggregated-ether-options lacp periodic fast
set interfaces ae1 aggregated-ether-options lacp system-id 01:01:01:01:01:01
set interfaces ae1 unit 0 family ethernet-switching interface-mode trunk
set interfaces ae1 unit 0 family ethernet-switching vlan members V388
Peer device
set interfaces ae2 aggregated-ether-options lacp active
set interfaces ae2 aggregated-ether-options lacp periodic fast
set interfaces ae2 unit 0 family ethernet-switching interface-mode trunk
set interfaces ae2 unit 0 family ethernet-switching vlan members V1141
set interfaces ae2 unit 0 family ethernet-switching vlan members V1135
set interfaces ae2 unit 0 family ethernet-switching vlan members V1115
The fix to this problem will be made available in a future Junos OS release.
Meanwhile, a couple of workarounds that can be used are as follows:
- Remove minimum link configuration in the ae bundle.
Delete interfaces ae1 aggregated-ether-options minimum-links
OR
- Disable the core isolation feature in EVPN. (Use
set protocols evpn no-core-isolation
.)