[EX/QFX] IRB interface cannot communicate with VME interface in a VC setup

Summary:

This article explains the behavior of a Virtual Chassis (VC) when the em0 interface on one member is connected to a network port on the other member and vice versa (reference diagram in the next section). It also explains the steps to overcome this behavior and considerations to make. This behavior is most likely to be seen on EX4600 devices and QFX5k series devices involved in VC.

Symptoms:

Consider a 2 member ( FPC 0 and FPC 1) Virtual Chassis scenario with FPC 0 as the master. Let us consider the network ports and em0/em1 connected as below:

 ---------                                   ---------
 | FPC 0 |---(em0)--------------(ge-1/0/4)---| FPC 1 |
 ---------                                   ---------
     |                                           |
     |                                           |
 (ge-0/0/4)------------------------------------(em1)

In this scenario, it is possible to observe that the IRB interface will not be able to communicate with the VME interface.

The MAC addresses assigned for multiple cases is as shown below:

Base Address:

root# run show chassis mac-addresses   
FPC 0
    Base address     12:12:12:12:a5:00
    Count            1280

IRB MAC Address:

[master:0][edit]
root# run show interfaces irb extensive | match hard
  Current address: 12:12:12:12:a5:00, Hardware address: 12:12:12:12:a5:00

VME MAC Address:

[master:0][edit] root# run show interfaces vme extensive | match hard Current address: 12:12:12:12:a5:01, Hardware address: 12:12:12:12:a5:01

Management Interface MAC:

[master:0][edit] root# run show interfaces em0 extensive | match hard Current address: 12:12:12:12:a5:01, Hardware address: 12:12:12:12:a9:f8

As we can see in the above outputs, IRB MAC address is the same as the Base address. This is not a favorable case and leads PFE to drop the packets.

Cause:

This issue is seen when the IRB MAC address takes the same MAC address as the Base address for the FPC.

Solution:

In order to overcome the situation, there has been a hidden-knob introduced for setups similar to the one depicted above. The irb-mac-shift knob under the packet-forwarding-options hierarchy directs the IRB MAC to change to "Base MAC + 2"

Example: Considering the outputs provided in the Symptoms section, the base address being 12:12:12:12:a5:00, the IRB MAC address changes to "base_address +2". The resulting IRB MAC becomes 12:12:12:12:a5:02.

Enable IRB MAC Shift:

[master:0][edit]
root# set system packet-forwarding-options irb-mac-shift

Commit the configuration change:

As you try to commit the configuration, the system indicates that a reboot is mandatory in order for the change to be reflected. Please make sure you are prepared for a reboot while you make this configuration change.

root# show system packet-forwarding-options               
irb-mac-shift;
 
[master:0][edit]
root# commit
[edit system]
  'packet-forwarding-options'
    warning: Reboot the system for the irb-mac-shift to take effect
configuration check succeeds
commit complete

Perform Reboot:

[master:0][edit]
root# run request system reboot
Reboot the system ? [yes,no] (no) yes
*** FINAL System shutdown message from root@ ***                            
System going down IMMEDIATELY

Once the device is up and running after a reboot, there should be no issues seen w.r.t the communication between IRB and VME interfaces.

Verify the IRB MAC:

root# run show interfaces irb extensive | match hard
  Current address: 12:12:12:12:a5:02, Hardware address: 12:12:12:12:a5:02
    Flags: Hardware-Down Up SNMP-Traps 0x4004000 Encapsulation: ENET2
 
[master:0][edit]
root# run show interfaces vme extensive | match hard
  Current address: 12:12:12:12:a5:01, Hardware address: 12:12:12:12:a5:01
 
[master:0][edit]
root# run show interfaces em0 extensive | match hard
  Current address: 12:12:12:12:a5:01, Hardware address: 12:12:12:12:a9:f8

***IMPORTANT: This knob is only necessary for this specific network setting where management interface and network ports are connected in the form of a ring in VC formation. In other scenarios, configuring this is NOT recommended. Consult JTAC if you want to make sure about the need of this configuration as per your network setup***

Notes:

The management interface of the switch could be em/me ( Here, the document contains em0/em1 for demonstration purpose)
The hidden knob can be used in the following releases: junos:14.1X53-D47, junos:17.1R3, junos:17.2R3, junos:17.3R4, junos:17.4R2, junos:18.1R3, junos:18.2R2, junos:18.3R1.
The "irb-mac-shift" knob does not modify VME interface MAC and network port addresses.
This knob makes sense only for regular system upgrades (not for NSSU/ISSU) as a reboot is required for the change to take effect.
The knob is of hidden nature. Using a tab key for completion will not auto-populate the knob. Please type the entire knob manually to avoid errors.

Knowledge Base