In a Layer 3 VPN topology, executing a traceroute to a remote customer edge (CE) network may result in an Internet Control Message Protocol (ICMP) time to live (TTL) expired reply with a different source address than that of the correct exit interface IP address as per the routing table.

This article explains that this is the default behavior of traceroute in an L3VPN setup on Junos OS with different interface cards, and details the Junos OS releases in which this default behavior can be modified.

 

Consider the following L3VPN topology:​

 

In PE1, a Modular Port Concentrator (MPC) Type 2 card is used to connect CE-A.

PE1> show chassis hardware | match MPC                                 
FPC 1    REV 31   750-031089   CACS1064     MPC Type 2 3D

 

In PE2, a Dense Port Concentrator (DPC) card is used to connect CE-B.

PE2> show chassis hardware | match DPC             
FPC 2 REV 22   750-022366   ZG6375     DPCE 20x 1GE + 2x 10GE R EQ

 

traceroute to a remote CE-B from CE-A results in the correct exit interface IP address in PE2 with the DPC line card.

CE-A> traceroute 24.24.24.24

traceroute to 24.24.24.24 (24.24.24.24), 30 hops max, 40 byte packets
 1  11.11.11.2 (11.11.11.2)  1.661 ms  1.598 ms  0.973 ms
 2  * * *
 3  20.20.20.2 (20.20.20.2)  4.310 ms  0.843 ms  0.681 ms
 4  24.24.24.24 (24.24.24.24)  1.202 ms  1.282 ms  1.159 ms​

PE2> show route table vrf-a.inet.0  

vrf-a.inet.0: 9 destinations, 9 routes (9 active, 0 hold down, 0 hidden)
+ = Active Route, - = Last Active, * = Both
24.24.24.24/32     *[OSPF/10] 00:03:37, metric 1
                    > to 20.20.20.1 via ge-2/1/1.0
                      to 21.21.21.1 via ge-2/1/0.0

PE2> show interfaces terse routing-instance vrf-a   

Interface               Admin Link Proto    Local                 Remote
ge-2/1/0.0              up    up   inet     21.21.21.2/30  
                                   multiservice
ge-2/1/1.0              up    up   inet     20.20.20.2/30  
                                   multiservice
lsi.2                   up    up   inet   
                                   iso    
                                   inet6  

 

However, traceroute to a remote CE-A from CE-B results in an incorrect exit interface IP address in PE1 with the MPC Type 2 card.

CE-B> traceroute 14.14.14.14   

traceroute to 14.14.14.14 (14.14.14.14), 30 hops max, 40 byte packets
 1  20.20.20.2 (20.20.20.2)  0.917 ms  0.791 ms  0.650 ms
 2  * * *
 3  12.12.12.2 (12.12.12.2)  1.282 ms  2.741 ms  3.048 ms
 4  14.14.14.14 (14.14.14.14)  1.203 ms  1.316 ms  1.165 ms​

vrf-a.inet.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

14.14.14.14/32     *[OSPF/10] 00:00:58, metric 1
                    > to 11.11.11.1 via ge-1/0/2.0
                      to 12.12.12.1 via ge-1/0/3.0

PE1> show interfaces terse routing-instance vrf-a 

Interface               Admin Link Proto    Local                 Remote
ge-1/0/2.0              up    up   inet     11.11.11.2/30   
                                   multiservice
ge-1/0/3.0              up    up   inet     12.12.12.2/30   
                                   multiservice
lsi.1                   up    up   inet    
                                   iso     

 

This is expected behavior with the MPC line card in PE1 where the first configured interface under vrf-instance is used.

• The behavior in DPC or an earlier chipset is for the ICMP TTL expired packets to use the source address as the address of the exit interface as per the routing table, if a virtual routing and forwarding (VRF) instance is configured with vrf-table-label.

• In MPC line cards, however, the behavior is different and the first configured interface under vrf-instance is used.

Note: This scenario is applicable only when there are two or more CE-facing interfaces from PE.

 

In Junos OS 15.1R5 and later releases (PR1358376), there is an option to change this default behavior by using the kernel sysctl parameter called icmp.traceroute_l3vpn.

Setting this parameter value to "1" will enable selection of the correct exit interface IP address as per the routing table.

> start shell user root
% sysctl -w net.inet.icmp.traceroute_l3vpn=1

 

To change the kernel sysctl parameter value to “1” in PE1, use the following:

pe1-re0> start shell user root
Password:

root@pe1-re0:/var/home/labroot # sysctl -w net.inet.icmp.traceroute_l3vpn=1
net.inet.icmp.traceroute_l3vpn: 0 -> 1

root@pe1-re0:/var/home/labroot #exit
pe1-re0>

 

traceroute to the remote CE-A from CE-B results in the selection of the correct exit interface IP address after changing the sysctl parameter value to "1".

CE-B> traceroute 14.14.14.14

traceroute to 14.14.14.14 (14.14.14.14), 30 hops max, 40 byte packets
 1  20.20.20.2 (20.20.20.2)  6.074 ms  0.791 ms  0.684 ms
 2  * * *
 3  11.11.11.2 (11.11.11.2)  1.313 ms  1.118 ms  1.001 ms
 4  14.14.14.14 (14.14.14.14)  1.194 ms  1.299 ms  1.158 ms

 

Note: The sysctl parameter value reverts to "0" following a system reboot.