This article explains the behavior of a Virtual Chassis (VC) when the em0 interface on one member is connected to a network port on the other member and vice versa (reference diagram in the next section). It also explains the steps to overcome this behavior and considerations to make. This behavior is most likely to be seen on EX4600 devices and QFX5k series devices involved in VC.
Consider a 2 member ( FPC 0 and FPC 1) Virtual Chassis scenario with FPC 0 as the primary. Let us consider the network ports and em0/em1 connected as below:
--------- ---------
| FPC 0 |---(em0)--------------(ge-1/0/4)---| FPC 1 |
--------- ---------
| |
| |
(ge-0/0/4)------------------------------------(em1)
In this scenario, it is possible to observe that the IRB interface will not be able to communicate with the VME interface.
The MAC addresses assigned for multiple cases is as shown below:
Base Address:
root# run show chassis mac-addresses
FPC 0
Base address 12:12:12:12:a5:00
Count 1280
IRB MAC Address:
[master:0][edit]
root# run show interfaces irb extensive | match hard
Current address: 12:12:12:12:a5:00, Hardware address: 12:12:12:12:a5:00
VME MAC Address:
[master:0][edit]
root# run show interfaces vme extensive | match hard
Current address: 12:12:12:12:a5:01, Hardware address: 12:12:12:12:a5:01
Management Interface MAC:
[master:0][edit]
root# run show interfaces em0 extensive | match hard
Current address: 12:12:12:12:a5:01, Hardware address: 12:12:12:12:a9:f8
As we can see in the above outputs, IRB MAC address is the same as the Base address. This is not a favorable case and leads PFE to drop the packets.
This issue is seen when the IRB MAC address takes the same MAC address as the Base address for the FPC.
In order to overcome the situation, there has been a hidden-knob introduced for setups similar to the one depicted above. The
irb-mac-shift knob under the
packet-forwarding-options hierarchy directs the IRB MAC to change to "Base MAC + 2"
Example: Considering the outputs provided in the Symptoms section, the base address being 12:12:12:12:a5:00, the IRB MAC address changes to "base_address +2". The resulting IRB MAC becomes 12:12:12:12:a5:02.
Enable IRB MAC Shift:
[master:0][edit]
root# set system packet-forwarding-options irb-mac-shift
Commit the configuration change:
As you try to commit the configuration, the system indicates that a reboot is mandatory in order for the change to be reflected. Please make sure you are prepared for a reboot while you make this configuration change.
root# show system packet-forwarding-options
irb-mac-shift;
[master:0][edit]
root# commit
[edit system]
'packet-forwarding-options'
warning: Reboot the system for the irb-mac-shift to take effect
configuration check succeeds
commit complete
Perform Reboot:
[master:0][edit]
root# run request system reboot
Reboot the system ? [yes,no] (no) yes
*** FINAL System shutdown message from root@ ***
System going down IMMEDIATELY
Once the device is up and running after a reboot, there should be no issues seen w.r.t the communication between IRB and VME interfaces.
Verify the IRB MAC:
root# run show interfaces irb extensive | match hard
Current address: 12:12:12:12:a5:02, Hardware address: 12:12:12:12:a5:02
Flags: Hardware-Down Up SNMP-Traps 0x4004000 Encapsulation: ENET2
[master:0][edit]
root# run show interfaces vme extensive | match hard
Current address: 12:12:12:12:a5:01, Hardware address: 12:12:12:12:a5:01
[master:0][edit]
root# run show interfaces em0 extensive | match hard
Current address: 12:12:12:12:a5:01, Hardware address: 12:12:12:12:a9:f8
***IMPORTANT: This knob is only necessary for this specific network setting where management interface and network ports are connected in the form of a ring in VC formation. In other scenarios, configuring this is NOT recommended. Consult JTAC if you want to make sure about the need of this configuration as per your network setup***
Notes:
- The management interface of the switch could be em/me ( Here, the document contains em0/em1 for demonstration purpose)
- The hidden knob can be used in the following releases: junos:14.1X53-D47, junos:17.1R3, junos:17.2R3, junos:17.3R4, junos:17.4R2, junos:18.1R3, junos:18.2R2, junos:18.3R1.
- The "
irb-mac-shift" knob does not modify VME interface MAC and network port addresses.
- This knob makes sense only for regular system upgrades (not for NSSU/ISSU) as a reboot is required for the change to take effect.
- The knob is of hidden nature. Using a tab key for completion will not auto-populate the knob. Please type the entire knob manually to avoid errors.
2021-03-24: Updated the article terminology to align with Juniper's Inclusion & Diversity initiatives