Support Support Downloads Knowledge Base Juniper Support Portal Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Contrail] Understanding CPU pinning in DPDK configuration



Article ID: KB35517 KB Last Updated: 02 Mar 2020Version: 1.0

In a DPDK compute node, a common question that users ask is why does the dpdk-vrouter have an affinity to all available CPU cores in the system even if it is configured to bind to specific cores in the config file. This article explains this behavior.  

From the configuration below, we pin CPU cores 1, 25, 2, 26, 13, 37, 14, and 38 for DPDK module:
    $ cat /etc/contrail/supervisord_vrouter_files/contrail-vrouter-dpdk.ini|grep taskset
        -c 1,25,2,26,13,37,14,38 /usr/bin/contrail-vrouter-dpdk
        --vdev "eth_bond_bond0,mode=4,xmit_policy=l34,socket_id=1,mac=14:02:ec:66:b8:dc,slave=0000:87:00.0,slave=0000:09:00.1"
        --vlan_tci "722" 
        --vlan_fwd_intf_name "bond0" 
        --socket-mem 1024,1024

Note: The output gives one long line which has been broken into multiple lines for better readability.

From the output shown above, the expected result should be dpdk-vrouter with CPU affinity set to `1,25,2,26,13,37,14,38`. However, the DPDK thread uses more than those 8 defined cores:

    $ ps -o pid,spid,comm,cpuid -Tp `pidof contrail-vrouter-dpdk`
    10416 10416 contrail-vroute    21
    10416 10665 eal-intr-thread     0
    10416 10666 lcore-slave-1       0
    10416 10667 lcore-slave-2      12
    10416 10668 lcore-slave-8      17
    10416 10669 lcore-slave-9      20
    10416 10670 lcore-slave-10      1
    10416 10671 lcore-slave-11      2
    10416 10672 lcore-slave-12     13
    10416 10673 lcore-slave-13     14
    10416 10674 lcore-slave-14     25
    10416 10675 lcore-slave-15     26
    10416 10676 lcore-slave-16     37
    10416 10677 lcore-slave-17     38
    10416 13674 lcore-slave-9      15

The `ps` command first calls another command, `pidof contrail-vrouter-dpdk` to get the process ID (PID) of DPDK process. With "thread view" (`-T`) it prints information about each thread in the same process separately. The `-o` option specifies what columns to look at. In this example, it will print four columns: 

* `PID`: process ID of current dpdk process
* `SPID`: thread process ID of each threads
* `comm`: name of each thread

In this setup, a total of 15 CPU cores are being used by DPDK threads. Furthermore, with `taskset` command, we can see that the DPDK process indeed utilizes all 48 cores in the system.

    $ taskset -c -p `pidof contrail-vrouter-dpdk`
    pid 10416's current affinity list: 0-47

This behavior is working as designed. With the following `-a` option added in the same `taskset` command, from the output, we can see that cores for forwarding are using what is defined in the configuration file `contrail-vrouter-dpdk.ini`. However, for control we can use any core from 0-47.

    $ taskset -a -cp `pidof contrail-vrouter-dpdk `
    pid 10416's current affinity list: 0-47     #<---control
    pid 10665's current affinity list: 0-47     #<---control
    pid 10666's current affinity list: 0-47     #<---control
    pid 10667's current affinity list: 0-47     #<---control
    pid 10668's current affinity list: 0-47     #<---control
    pid 10669's current affinity list: 0-47     #<---control
    pid 10670's current affinity list: 1        #<---forwarding
    pid 10671's current affinity list: 2        #<---forwarding
    pid 10672's current affinity list: 13       #<---forwarding
    pid 10673's current affinity list: 14       #<---forwarding
    pid 10674's current affinity list: 25       #<---forwarding
    pid 10675's current affinity list: 26       #<---forwarding
    pid 10676's current affinity list: 37       #<---forwarding
    pid 10677's current affinity list: 38       #<---forwarding
    pid 13674's current affinity list: 0-47     #<---forwarding

The output of the `ps` command lists all the cores used by DPDK, including cores used both for forwarding and control purposes.  However, the `contrail-vrouter-dpdk.ini` configuration file only defines the cores allowed
to be used for forwarding purpose. In other words, the CPU pinning for `contrail-vrouter-dpdk` process is only for resource consuming threads that forward packets. The pinning doesn't apply to the threads doing control related tasks. To change this default behavior,  use taskset to pin the control threads if needed. Or pin the CPUs to certain VMs to prevent those threads from using the same CPUs.

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search