This article applies to QFX5K as LSR device. For MPLS packets, QFX5K platform considers IP MTU of the egress interface when putting the MPLS packet on the wire.
MPLS Packets with MTU size more that INET MTU will be dropped as QFX5k will not calculate MPLS family MTU separately instead will use INET MTU for MPLS Packets as well.
Example:
Packet Drops at size 9073 when INET MTU is 9100 though MPLS is set/calculated at larger value at 9158.
root@Switch> ping 30.30.30.30 size 9073 do-not-fragment
PING 30.30.30.30 (30.30.30.30): 9073 data bytes
ping: sendto: Message too long
-------------------------------------------------
Logical interface et-0/0/32.0 (Index 572) (SNMP ifIndex 601)
Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
Input packets : 233447
Output packets: 233447
Protocol inet, MTU: 9100 <--- INET MTU
Max nh cache: 75000, New hold nh limit: 75000, Curr nh cnt: 1, Curr new hold cnt: 0, NH drop cnt: 0
Flags: Sendbcast-pkt-to-re, User-MTU
Addresses, Flags: Is-Preferred Is-Primary
Destination: 192.168.40/24, Local: 192.168.40.2, Broadcast: 192.168.40.255.
Protocol mpls, MTU: 9166, Maximum labels: 3. <--- Note the MTU and label size
Note : Even with the configuration of the mpls mtu 9178 the ping size does not exceed 9072 - details below
{master:0}[edit]
labroot@Switch# set interface xe-0/0/1 unit 0 family mpls mtu 9178 <--- Configure MPLS MTU as 9178
labroot@Switch> ping 30.30.30.30 size 9073 do-not-fragment
PING 30.30.30.30 (30.30.30.30): 9073 data bytes
ping: sendto: Message too long
According to the output above, QFX does calculate MPLS MTU as 9166 but it does not send updates to PFE. Hence, PFE will consider the MPLS MTU equivalent to INET MTU due to which MPLS packets above INET MTU will be dropped.
As a workaround, configure INET MTU for MPLS packets in addition to considering the number of labels (max 3).
Example:
If Interface MTU is configured as 9192, MPLS MTU is calculated reducing 14 bytes for Ethernet frame and 12 bytes reserved for MPLS (considering maximum label of 3) making it 9166.
root@QFX5k > ...9 extensive | match mtu
Link-level type: Ethernet, MTU: 9192, LAN-PHY mode, Speed: 100Gbps,
FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
Protocol inet, MTU: 9100, Generation: 588410521207, Route table: 0
Flags: User-MTU
Protocol mpls, MTU: 9166, Maximum labels: 3, Generation: 588410521208,
To avoid MPLS packet drops, configure INET MTU reducing 14 bytes for ethernet frame and 12 bytes for MPLS from the IFD MTU.
To configure the INET MTU use the following command:
labroot@Switch# set interface xe-0/0/* unit 0 family inet mtu #value# <--- Select a value based on the above calculation.