Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[vMX] Unable to connect SRIOV interface to physical CPE for vMX performance-mode

0

0

Article ID: KB36486 KB Last Updated: 12 Mar 2021Version: 1.0
Summary:

When vBNG is deployed on RHOSP13 cluster with vMX (performace-mode) SRIOV NIC, the vMX SRIOV interface is not connecting properly to the physical CPE (with single/double vlan-tags). 

In this example:
  • Physical CPE's are connected via Openstack SRIOV Network ('vbng-cpe-sriov') and coming to vMX SRIOV interface with stacked vlan [outer:3321 inner:100,10]
  • 2 CPEs configured to generate traffic with single vlan tag 100 & 101. Outer vlan is added by the L2 Switch which is connecting the CPEs to vBNG SRIOV interface.
  • DC Gateway is connected to vMX VirtIO interface via Openstack VirtIO network 'vbng-int-virtio' with single vlan tag 3320.
Symptoms:

Topology:

DC Gateway<--->([vlan:3320]ge-0/0/2) MX (ge-0/0/3[vlan:3321.x)<--->(q-in-q)L2 switch<--->Physical CPE(IPoE/PPPoE Subs.)
  • ge-0/0/2 : Is configured in VirtIO mode connecting to Openstack network 'vbng-int-virtio' (vlan:3320, connected via openstack's internal OVS) 
  • ge-0/0/3 : Is configured in SRIOV mode connecting to Openstack network 'vbng-cpe-sriov' (vlan:3321, connected via VF function of SRIOV NIC port)

Note:

Physical CPEs are generating traffic with single tag (vlan 100,101) & L2-Switch is adding Outer tag 3321. So When traffic reaches to vMX SRIOV port, it's q-in-q (stacked vlan) for IPoE/PPPoE Subscriber.   

Configuration:

interfaces {
    ge-0/0/2 {
        inactive: hierarchical-scheduler maximum-hierarchy-levels 2;
        flexible-vlan-tagging;
        speed 10g;
        unit 3320 {
            vlan-id 3320;
            family inet {
                address 10.0.31.10/28;
            }
            family inet6 {
                address 2004:2004::1/64;
            }
        }
    }
    ge-0/0/3 {
        inactive: hierarchical-scheduler maximum-hierarchy-levels 2;
        flexible-vlan-tagging;
        vlan-offload;
        speed 10g;
        auto-configure {
            stacked-vlan-ranges {
                dynamic-profile AUTO-VLAN-STACK {
                    accept [ dhcp-v4 dhcp-v6 ];
                    ranges {
                        3000-4000,any;
                    }
                }
            }
        }
        encapsulation flexible-ethernet-services;
    }
}

Core Interface ge-0/0/2 (connecting to DC Gateway) is working fine with virtio (openstack network 'vbng-int-virtio')

root@vmx-v1> show route 10.0.31.1

inet.0: 27 destinations, 27 routes (27 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.31.0/28       *[Direct/0] 5d 01:56:18
                    >  via ge-0/0/2.3320

root@vmx-v1> show ospf neighbor
Address          Interface              State           ID               Pri  Dead
10.0.31.1        ge-0/0/2.3320          Full            10.240.32.128      1    31

root@vmx-v1> ping 10.0.31.1
PING 10.0.31.1 (10.0.31.1): 56 data bytes
64 bytes from 10.0.31.1: icmp_seq=0 ttl=64 time=34.395 ms
64 bytes from 10.0.31.1: icmp_seq=1 ttl=64 time=16.894 ms
64 bytes from 10.0.31.1: icmp_seq=2 ttl=64 time=11.529 ms
^C
--- 10.0.31.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 11.529/20.939/34.395/9.763 ms


But with SRIOV interface (ge-0/0/3), there are No IPoE/PPPoE subscribers. The CPE is UP and vlans are properly configured in L2-Switch for the IPoE/PPPoE traffic to hit VMX SRIOV interface with 3321.x vlan tagging.
On further troubleshooting with packet capture on ge-0/0/3, there are no packets captured with 3321 Vlan ID on ge-0/0/3.

root@vmx-v1> show subscribers
Interface             IP Address/VLAN ID                      User Name                      LS:RI

root@vmx-v1>
Cause:

The SRIOV NICs are installed with native Intel i40e driver instead of Juniper provided modified SRIOV driver. For Native Intel SRIOV Driver, use 'set interface <interface-name> vlan-offload' command to offload the VLAN filtering to unmodified PF (Intel stock) driver.

Note: vMX was built using HEAT Template in RHOSP13 platform, and each compute node in RHOSP13 cluster has 2 Intel SRIOV NIC (XXV710 - 25GbE SFP28) p1p1 & p2p1 with 64 VFs per NIC.

[root@test-compute-1]# ethtool -i p2p1
driver: i40e
version: 2.3.2-k
firmware-version: 6.00 0x80003785 18.5.17
expansion-rom-version:
bus-info: 0000:5e:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
[root@test-compute-1]# lspci -nn | grep Ether
01:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
01:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
19:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)
19:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)
5e:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 [8086:158b] (rev 02)
5e:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 [8086:158b] (rev 02)
5e:02.0 Ethernet controller [0200]: Intel Corporation Ethernet Virtual Function 700 Series [8086:154c] (rev 02)
. . .
5e:09.2 Ethernet controller [0200]: Intel Corporation Ethernet Virtual Function 700 Series [8086:154c] (rev 02)   <--SRIOV Port of vMX(ge-0/0/3)
. . .
5e:09.7 Ethernet controller [0200]: Intel Corporation Ethernet Virtual Function 700 Series [8086:154c] (rev 02)
d8:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 [8086:158b] (rev 02)
d8:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 [8086:158b] (rev 02)

[root@test-compute-1]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
. . .
8: p2p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 3c:fd:fe:bf:83:e0 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    . . .
    vf 58 MAC fa:16:3e:5f:0c:c5, vlan 3321, spoof checking off, link-state enable, trust off    <--SRIOV Port of vMX(ge-0/0/3)
    . . .

    vf 63 MAC c6:2d:13:33:87:2a, spoof checking off, link-state auto, trust off

[root@test-compute-1]# virsh dumpxml instance-000009ba

<domain type='kvm' id='29'>
  <name>instance-000009ba</name>
  <uuid>$ABC123</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="17.0.7-2.el7ost"/>
      <nova:name>Instance_RSP_In_A_Box-vmx-Test_fpc0</nova:name>
    . . .
    <interface type='vhostuser'>
      <mac address='fa:16:3e:fa:9a:73'/>
      <source type='unix' path='/var/lib/vhost_sockets/vhua1c82c0a-d9' mode='server'/>
      <target dev='vhua1c82c0a-d9'/>
      <model type='virtio'/>
      <driver rx_queue_size='1024' tx_queue_size='1024'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <interface type='hostdev' managed='yes'>
      <mac address='fa:16:3e:5f:0c:c5'/>
      <driver name='vfio'/>                   <--SRIOV Interface ge-0/0/3
      <source>
        <address type='pci' domain='0x0000' bus='0x5e' slot='0x09' function='0x2'/>
      </source>
      <vlan>
        <tag id='3321'/>
      </vlan>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>
    . . .
Solution:

To solve this problem, add the 'vlan-offload' command under ge-0/0/3 (SRIOV) interface as below, then commit.

root@vmx-v1# set interfaces ge-0/0/3 ?
. . .
  vlan-offload         Offload VLAN filtering to NIC HW

[edit]
root@vmx-v1# set interfaces ge-0/0/3 vlan-offload
root@vmx-v1# commit and-quit


Final Configuration:

interfaces {
    ge-0/0/2 {
        inactive: hierarchical-scheduler maximum-hierarchy-levels 2;
        flexible-vlan-tagging;
        speed 10g;
        unit 3320 {
            vlan-id 3320;
            family inet {
                address 10.0.31.10/28;
            }
            family inet6 {
                address 2004:2004::1/64;
            }
        }
    }
    ge-0/0/3 {
        inactive: hierarchical-scheduler maximum-hierarchy-levels 2;
        flexible-vlan-tagging;
        vlan-offload;
        speed 10g;
        auto-configure {
            stacked-vlan-ranges {
                dynamic-profile AUTO-VLAN-STACK {
                    accept [ dhcp-v4 dhcp-v6 ];
                    ranges {
                        3000-4000,any;
                    }
                }
            }
        }
        encapsulation flexible-ethernet-services;
    }
    lo0 {
        unit 0 {
            family inet {
                address 10.1.100.1/32;
            }
            family inet6 {
                address 2004::1/128;
            }
        }
    }
}

root@vmx-v1> show route 10.0.31.1

inet.0: 27 destinations, 27 routes (27 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.31.0/28       *[Direct/0] 5d 02:19:34
                    >  via ge-0/0/2.3320

root@vmx-v1> show ospf neighbor
Address          Interface              State           ID               Pri  Dead
10.0.31.1        ge-0/0/2.3320          Full            10.240.32.128      1    33

root@vmx-v1> ping 10.0.31.1
PING 10.0.31.1 (10.0.31.1): 56 data bytes
64 bytes from 10.0.31.1: icmp_seq=0 ttl=64 time=73.270 ms
64 bytes from 10.0.31.1: icmp_seq=1 ttl=64 time=4.010 ms
^C
--- 10.0.31.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 4.010/38.640/73.270/34.630 ms

root@vmx-v1> show subscribers
Interface             IP Address/VLAN ID                      User Name                      LS:RI
demux0.3221225472     0x8100.3321 0x8100.100                                            default:default
demux0.3221225476     0x8100.3321 0x8100.101                                            default:default

root@vmx-v1> show subscribers extensive
Type: VLAN
Logical System: default
Routing Instance: default
Interface: demux0.3221225472
Interface type: Dynamic
Underlying Interface: ge-0/0/3
Dynamic Profile Name: AUTO-VLAN-STACK
Dynamic Profile Version: 1
State: Active
Session ID: 1
PFE Flow ID: 13
Stacked VLAN Id: 0x8100.3321
VLAN Id: 0x8100.100
Login Time: 2021-01-20 00:01:38 UTC

Type: VLAN
Logical System: default
Routing Instance: default
Interface: demux0.3221225476
Interface type: Dynamic
Underlying Interface: ge-0/0/3
Dynamic Profile Name: AUTO-VLAN-STACK
Dynamic Profile Version: 1
State: Active
Session ID: 5
PFE Flow ID: 19
Stacked VLAN Id: 0x8100.3321
VLAN Id: 0x8100.101
Login Time: 2021-01-20 05:24:12 UTC
Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search