Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Contrail] DPDK successfully loads but vRouter-agent is stuck in initializing state

0

0

Article ID: KB33830 KB Last Updated: 13 Feb 2019Version: 1.0
Summary:

DPDK binary has been loaded successfully, but Contrail vrouter-agent is stuck in initializing state. The vhost0 is not reachable from the configured gateway, and if the gateway sends broadcast traffic such as ARP request, it is seen on the vhost0's packet sniffer but unicast packet from gateway such as ICMP request, is not received by vhost0. 

​This article explains what to look for you encounter this failure state and how to resolve the issue.

Symptoms:

Running 'contrail-status' shows following state:

# contrail-status -d
== Contrail vRouter ==
supervisor-vrouter:           active
contrail-vrouter-agent        initializing (XMPP:control-node:172.18.101.102, XMPP:control-node:172.18.101.103, XMPP:dns-server:172.18.101.101, XMPP:dns-server:172.18.101.103, Collector connection down)pid 12194, uptime 4 days, 17:18:01     
contrail-vrouter-dpdk         active              pid 12080, uptime 4 days, 17:18:04     
contrail-vrouter-nodemgr      initializing (Collector, Discovery:Collector[Subscribe - Connection Error] connection down)pid 12079, uptime 4 days, 17:18:04   
 
# vifdump -i 0 arp and host 172.29.21.195
vif0/0      PCI: 0000:00:00.0 (Speed 10000, Duplex 1)
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on mon0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:25:32.591831 ARP, Request who-has 172.29.21.195 tell 172.29.21.214, length 46
18:25:33.595801 ARP, Request who-has 172.29.21.195 tell 172.29.21.214, length 46

The vhost0 is not reachable from the gateway or to the gateway. Traffic capture done on bond0 or vhost0 confirms ARP request from the gateway was received, but no ARP response is sent back.

Cause:

The most common cause is due to a wrong MAC being configured for DPDK module, which is not the same as the vRouter agent's configured MAC or bond0/vhost0's MAC address.

Solution:
  1. Check the exact MAC used by DPDK module in the memory.

    Example with mac=14:02:ec:6b:5f:4c used by DPDK:

    # ps -aux | grep dpdk   
    root     11472  709  0.0 4091000 7140 ?        Sl   17:16  41:01 /usr/bin/contrail-vrouter-dpdk --no-mrgbuf --no-daemon --vr_mpls_labels 6000 --vr_flow_entries=2097152 --vdev eth_bond_bond0,mode=4,xmit_policy=l34,socket_id=0,mac=14:02:ec:6b:5f:4c,slave=0000:08:00.0,slave=0000:81:00.1 --vr_mpls_labels=6000 --vlan_tci 2004 --vlan_fwd_intf_name bond0 --socket-mem 1024 1024
    root     17316  0.0  0.0  10476   908 pts/3    S+   17:22   0:00 grep dpdk
  2. The above MAC must be consistent with vRouter agent's MAC and therefore vhost0's MAC. In the following output, we can see it is different:

    # ifconfig vhost0
    vhost0    Link encap:Ethernet  HWaddr 00:11:ac:1d:d4:58 
              inet addr:172.29.21.214  Bcast:172.29.21.255  Mask:255.255.255.192
              inet6 addr: fe80::686e:20ff:fefe:6e63/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:9100  Metric:1
              RX packets:1554 errors:0 dropped:196 overruns:0 frame:0
              TX packets:539 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:95802 (95.8 KB)  TX bytes:67274 (67.2 KB)
    ​ ​​
    This is because the user may have entered a wrong MAC in DPDK's initial file.
    cat /etc/contrail/supervisord_vrouter_files/contrail-vrouter-dpdk.ini
    [program:contrail-vrouter-dpdk]
    command=taskset -c  1,25,2,26,13,37,14,38 /usr/bin/contrail-vrouter-dpdk --no-mrgbuf --no-daemon --vr_mpls_labels 6000 --vr_flow_entries=2097152 --vdev "eth_bond_bond0,mode=4,xmit_policy=l34,socket_id=0,mac=14:02:ec:6b:5f:4c,slave=0000:08:00.0,slave=0000:81:00.1" --vr_mpls_labels=6000 --vlan_tci "2004" --vlan_fwd_intf_name "bond0" --socket-mem 1024,1024
    priority=410
    autostart=True
    killasgroup=True
    stdout_capture_maxbytes=1MB
    redirect_stderr=True
    stdout_logfile=/var/log/contrail/contrail-vrouter-dpdk-stdout.log
    stderr_logfile=/var/log/contrail/contrail-vrouter-dpdk-stderr.log
    exitcodes=0
    loglevel=debug​

After fixing the error above and restarting supervisor-vrouter, the vrouter agent comes up as expected. 

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search