Multicast upstream interface showing wrong interface after a routing change

Summary:

Multicast upstream interface showing wrong interface after a routing change. The issue is seen when the group-ranges have /32 prefix instead of a subnet. for example: set routing-instances JTAC-INSTANCE protocols pim rp static address 10.0.103.9 group-ranges 239.100.10.4/32. Upstream traffic will not be switched to the new IIF on a path changeover and that will cause traffic to be impacted.

Symptoms:

Here is an example of the issue:

Before triggering the issue, the uplink interface is via ae2.1004. There is also a backup path which is via irb.1099.

root@spine1> show multicast route instance JTAC-INSTANCE group 239.100.10.4 source-prefix 10.100.100.6 extensive    
Instance: JTAC-INSTANCE Family: INET

Group: 239.100.10.4
    Source: 10.100.100.6/32
    Upstream interface: ae2.1004    
    Downstream interface list: 
        xe-0/0/31:1.4000
    Number of outgoing interfaces: 1
    Session description: Organisational Local Scope
    Statistics: 574 kBps, 585 pps, 43353 packets
    Next-hop ID: 2097151
    Upstream protocol: PIM
    Route state: Active
    Forwarding state: Forwarding
    Cache lifetime/timeout: 360 seconds
    Wrong incoming interface notifications: 21
    Uptime: 00:00:10

Trigger the routing change by disabling the uplink interface on the peer device

{master:0}
root@spine1> show multicast route instance JTAC-INSTANCE group 239.100.10.4 source-prefix 10.100.100.6 extensive    
Instance: JTAC-INSTANCE Family: INET

Group: 239.100.10.4
    Source: 10.100.100.6/32
    Upstream interface: ae2.1004    <<<<<<<<<<<<< incorrect upstream interface    
    Downstream interface list: 
        xe-0/0/31:1.4000
    Number of outgoing interfaces: 1
    Session description: Organisational Local Scope
    Statistics: 574 kBps, 585 pps, 43353 packets
    Next-hop ID: 2097151
    Upstream protocol: PIM
    Route state: Active
    Forwarding state: Forwarding
    Cache lifetime/timeout: 360 seconds
    Wrong incoming interface notifications: 21
    Uptime: 00:05:52

{master:0}
root@spine1> show route 10.100.100.6                                                                          

inet.0: 32 destinations, 34 routes (32 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[Static/5] 2w1d 02:39:27
                    > to 10.85.173.1 via em0.0
                    [OSPF/150] 1w6d 19:55:56, metric 0, tag 0
                    > to 172.31.0.229 via irb.996

JTAC-INSTANCE.inet.0: 36 destinations, 47 routes (36 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.100.100.0/24    *[OSPF/150] 5d 07:08:48, metric 0, tag 0
                    > to 10.5.2.165 via irb.1099

The correct output should be as below is below

{master:0} root@spine1> show pim join instance JTAC-INSTANCE extensive 239.100.10.4 source 10.100.100.6 Instance: PIM.JTAC-INSTANCE Family: INET R = Rendezvous Point Tree, S = Sparse, W = Wildcard Group: 239.100.10.4 Source: 10.100.100.6 Flags: sparse,spt Upstream interface: irb.1099 <<<<<<<<<<<<<Correct upstream interface Upstream neighbor: 10.5.2.165 Upstream state: Join to Source, No Prune to RP Keepalive timeout: 345 Uptime: 00:05:48 Downstream neighbors: Interface: Pseudo-GMP xe-0/0/31:1.4000 Number of downstream interfaces: 1 Number of downstream neighbors: 0

Cause:

This is because the /32 prefix is interpreted differently in the routing pipeline and it discards the data traffic received on the new IIF at the PFE stage instead of trapping to the CPU. So, IIF_MISMATCH mechanism can not be detected here with this config, resulting is traffic loss until the PIM join is timed out. {master:0} root@spine1> show multicast statistics instance JTAC-INSTANCE interface irb.1099 Instance: JTAC-INSTANCE Family: INET Interface: irb.1099 Routing protocol: PIM Mismatch error: 0 Mismatch: 0 Mismatch no route: 0 Kernel resolve: 0 Routing notify: 0 Resolve no route: 0 Resolve error: 0 Resolve filtered: 0 Notify filtered: 0 In kbytes: 0 In packets: 0 Out kbytes: 0 Out packets: 0

Solution:

Instead of /32, the customer configuration can be slightly tweaked to "set routing-instances JTAC-INSTANCE protocols pim rp static address 10.0.103.9 group-ranges 239.100.10.0/24" /24 subnet or any other non /32 prefix which can cover the required group range(s). Here is an example:

1) Remove the 239.100.10.4/32 and add 239.100.10.0/24 from spine1 pim rp configuration 

{master:0}
root@spine1> configure 
Entering configuration mode

{master:0}[edit]
root@spine1# delete routing-instances JTAC-INSTANCE protocols pim rp static address 10.0.103.9 group-ranges 239.100.10.4/32
{master:0}[edit]
root@spine1# set routing-instances JTAC-INSTANCE protocols pim rp static address 10.0.103.9 group-ranges 239.100.10.0/24  


2) Trigger the routing change by disabling the uplink interface on the peer device

3) Check the multicast route and pim join status for 239.100.10.4 

{master:0}
root@spine1> show pim join instance JTAC-INSTANCE extensive 239.100.10.4 source 10.100.100.6 
Instance: PIM.JTAC-INSTANCE Family: INET
R = Rendezvous Point Tree, S = Sparse, W = Wildcard

Group: 239.100.10.4
    Source: 10.100.100.6
    Flags: sparse,spt
    Upstream interface: irb.1099   <<<<<<<<<<<<<Correct upstream interface            
    Upstream neighbor: 10.5.2.165
    Upstream state: Join to Source, No Prune to RP
    Keepalive timeout: 322
    Uptime: 00:36:45 
    Downstream neighbors:
        Interface: Pseudo-GMP             
            xe-0/0/31:1.4000
    Number of downstream interfaces: 1
    Number of downstream neighbors: 0

{master:0}
root@spine1> show multicast route instance JTAC-INSTANCE group 239.100.10.4 source-prefix 10.100.100.6 extensive 
Instance: JTAC-INSTANCE Family: INET

Group: 239.100.10.4
    Source: 10.100.100.6/32
    Upstream interface: irb.1099  <<<<<<<<<<<<<Correct upstream interface    
    Downstream interface list: 
        xe-0/0/31:1.4000
    Number of outgoing interfaces: 1
    Session description: Organisational Local Scope
    Statistics: 982 kBps, 1000 pps, 54729 packets
    Next-hop ID: 2097151
    Upstream protocol: PIM
    Route state: Active
    Forwarding state: Forwarding
    Cache lifetime/timeout: 360 seconds
    Wrong incoming interface notifications: 1
    Uptime: 00:36:50

{master:0}
root@spine1> show route 10.100.100.6 

inet.0: 32 destinations, 34 routes (32 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[Static/5] 2w1d 02:29:52
                    > to 10.85.173.1 via em0.0
                    [OSPF/150] 1w6d 19:46:21, metric 0, tag 0
                    > to 172.31.0.229 via irb.996

content.inet.0: 36 destinations, 47 routes (36 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.100.100.0/24    *[OSPF/150] 5d 06:59:13, metric 0, tag 0
                    > to 10.5.2.165 via irb.1099

In the output, we see the IIF_MISMATCH incremented by 1 (which is cases by the routing change) PIM relies on PFE to send an IIF_MISMATCH on the new RPF and on this NEW mismatch Multicast route will be switched to the new RPF.

{master:0}
eng@spine1> show multicast statistics instance JTAC-INSTANCE interface irb.1099
Instance: JTAC-INSTANCE Family: INET
Interface: irb.1099
    Routing protocol:          PIM   Mismatch error:               0
    Mismatch:                    1   Mismatch no route:            0
    Kernel resolve:              0   Routing notify:               0
    Resolve no route:            0   Resolve error:                0
    Resolve filtered:            0   Notify filtered:              0
    In kbytes:                   0   In packets:                   0
    Out kbytes:                  0   Out packets:                  0

Knowledge Base