This article explains why a mismatched state may be seen for LACP bundles on QFX5K devices by demonstrating the working of the core isolation feature with LACP bundles in a setup with EVPN configuration. It also suggests a couple of workarounds that could be used to prevent the mismatched LACP bundle state.

Note: The problem was observed in Junos OS release 17.3R3-S2.

 

Consider the following topology:

 

Spines 01           spine 02
 
|                      |
|                      |
|                      |
Leaf 01 ---------------|
ae |
   |
   |
 Host

 

In the above topology, Leaf 01 on the QFX5100 device is connected to two spines. When the leaf loses connectivity to both spines, the EVPN core isolation feature is triggered and the LACP bundles (ae) are brought down (moving the Mux state to “Waiting”) so that no traffic enters Leaf 01 (basically isolating the QFX5100 device).

Note: By default, the core isolation feature is enabled on QFX5100 devices.

However, if the LACP configuration in the leaf has the minimum-links configuration enabled, the leaf will keep the ae bundle up and in the "collecting and distributing" state, a behavior that indicates a mismatched LACP bundle state. For on the peer device (works fine) as shown below, the state is “Attached,” which means no traffic is entering the leaf.

QFX5100 device showing collecting and distributing state

 

show lacp interfaces
Feb 11 17:36:50
Aggregated interface: ae1
    LACP state:       Role   Exp   Def  Dist  Col  Syn  Aggr  Timeout  Activity
      xe-0/0/4 CDN    Actor   No    No   Yes  Yes   No   Yes     Fast    Active
      xe-0/0/4 CDN  Partner   No    No    No   No  Yes   Yes     Fast    Active <<<<<<<<<<<<<<<
      xe-0/0/5 CDN    Actor   No    No   Yes  Yes   No   Yes     Fast    Active
      xe-0/0/5 CDN  Partner   No    No    No   No  Yes   Yes     Fast    Active
      xe-0/0/6 CDN    Actor   No    No   Yes  Yes   No   Yes     Fast    Active
      xe-0/0/6 CDN  Partner   No    No    No   No  Yes   Yes     Fast    Active
    LACP protocol:        Receive State  Transmit State          Mux State
      xe-0/0/4                  Current   Fast periodic Collecting distributing
      xe-0/0/5                  Current   Fast periodic Collecting distributing
      xe-0/0/6                  Current   Fast periodic Collecting distributing


Aggregated interface: ae2
    LACP state:       Role   Exp   Def  Dist  Col  Syn  Aggr  Timeout  Activity
      xe-0/0/17 CDN    Actor   No   No   Yes  Yes   No   Yes     Fast    Active
      xe-0/0/17 CDN  Partner   No   No    No   No  Yes   Yes     Fast    Active
      xe-0/0/18 CDN    Actor   No   No   Yes  Yes   No   Yes     Fast    Active
      xe-0/0/18 CDN  Partner   No   No    No   No  Yes   Yes     Fast    Active
      xe-0/0/19 CDN    Actor   No   No   Yes  Yes   No   Yes     Fast    Active
      xe-0/0/19 CDN  Partner   No   No    No   No  Yes   Yes     Fast    Active
    LACP protocol:        Receive State  Transmit State          Mux State
      xe-0/0/17                 Current   Fast periodic Collecting distributing
      xe-0/0/18                 Current   Fast periodic Collecting distributing
      xe-0/0/19                 Current   Fast periodic Collecting distributing

 

LACP peer showing attached state to indicate no traffic entering the leaf

show lacp interfaces ae2  
Aggregated interface: ae2
    LACP state:       Role   Exp   Def  Dist  Col  Syn  Aggr  Timeout  Activity
      xe-0/0/6       Actor    No    No    No   No  Yes   Yes     Fast    Active
      xe-0/0/6     Partner    No    No   Yes  Yes   No   Yes     Fast    Active
      xe-0/0/7       Actor    No    No    No   No  Yes   Yes     Fast    Active
      xe-0/0/7     Partner    No    No   Yes  Yes   No   Yes     Fast    Active
      xe-0/0/8       Actor    No    No    No   No  Yes   Yes     Fast    Active
      xe-0/0/8     Partner    No    No   Yes  Yes   No   Yes     Fast    Active
      xe-0/0/9       Actor    No    No    No   No  Yes   Yes     Fast    Active
      xe-0/0/9     Partner    No    No   Yes  Yes   No   Yes     Fast    Active
      xe-0/0/10      Actor    No    No    No   No  Yes   Yes     Fast    Active
      xe-0/0/10    Partner    No    No   Yes  Yes   No   Yes     Fast    Active
      xe-0/0/11      Actor    No    No    No   No  Yes   Yes     Fast    Active
      xe-0/0/11    Partner    No    No   Yes  Yes   No   Yes     Fast    Active
    LACP protocol:        Receive State  Transmit State          Mux State
      xe-0/0/6                  Current   Fast periodic           Attached
      xe-0/0/7                  Current   Fast periodic           Attached
      xe-0/0/8                  Current   Fast periodic           Attached

 

ae bundle configuration on both devices with minimum-links enabled

QFX

set interfaces ae1 esi 00:00:00:00:00:00:00:50:33:01
set interfaces ae1 esi all-active
set interfaces ae1 aggregated-ether-options minimum-links 2
set interfaces ae1 aggregated-ether-options lacp active
set interfaces ae1 aggregated-ether-options lacp periodic fast
set interfaces ae1 aggregated-ether-options lacp system-id 01:01:01:01:01:01
set interfaces ae1 unit 0 family ethernet-switching interface-mode trunk
set interfaces ae1 unit 0 family ethernet-switching vlan members V388

 

Peer device

set interfaces ae2 aggregated-ether-options lacp active
set interfaces ae2 aggregated-ether-options lacp periodic fast
set interfaces ae2 unit 0 family ethernet-switching interface-mode trunk
set interfaces ae2 unit 0 family ethernet-switching vlan members V1141
set interfaces ae2 unit 0 family ethernet-switching vlan members V1135
set interfaces ae2 unit 0 family ethernet-switching vlan members V1115

 

The fix to this problem will be made available in a future Junos OS release.

Meanwhile, a couple of workarounds that can be used are as follows:

1. Remove minimum link configuration in the ae bundle.

Delete interfaces ae1 aggregated-ether-options minimum-links

OR

2. Disable the core isolation feature in EVPN. (Use set protocols evpn no-core-isolation.)