Support Support Downloads Knowledge Base Juniper Support Portal Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Contrail] Trace openstack REST API message via packet capture

0

0

Article ID: KB33706 KB Last Updated: 08 Jan 2019Version: 2.0
Summary:

When troubleshooting contrail issues, it is often required to trace the REST API messages between different components with the following commonly used methods:

  • Use "--debug" option from OpenStack CLI
  • Enable debug level of traces in API logs
  • Use introspect whenever available
  • Capture packets on corresponding REST API TCP port

The first 3 methods are more convenient compared to the packet capture method. Unfortunately, they may not be supported in some cases. In contrast, you can always use tcpdump to capture packets between servers or even between processes in the same server. Therefore, it is an alternative troubleshooting method and can be used as the last resort when other methods are not supported or not applicable.

This article will demonstrate how to extract the REST API message content.

Solution:

In our setup, we have 3 contrail-api servers, each running on one node. For demonstration purposes, we shutdown one of the nodes so we only need to perform packet capture on the other two nodes. 

Use following tcpdump command to capture packets on each node:

on node "cont101": tcpdump -ni any port 9100 -w any9100.1.pcap
on node "cont103": tcpdump -ni any port 9100 -w any9100.3.pcap

This will capture: 

  • Packets interacting with contrail API service (port 9100) only
  • Packets from all interfaces, including:
    • management interface
    • data and control interface
    • loopback interface (lo)

With the above tcpdump running on the two nodes ("cont101" and "cont103" in this test), we'll run some tests to invoke contrail-api operations. Specifically, we'll attach a VMI into a VM, then detach it (see KB33691 - Attaching and detaching a VMI/port to a VM via OpenStack CLI for more details about this test). After this test, we'll have 2 pcap files generated each from one node. Then analyze the pcap file and find the REST API messages invoked by the VMI operations.  

The second tool is tshark:

tshark -r any9100.1.pcap -nY "http.request.full_uri contains virtual-machine" -t ud -w any9100.1.vmi.pcap
tshark -r any9100.3.pcap -nY "http.request.full_uri contains virtual-machine" -t ud -w any9100.3.vmi.pcap

The above tshark commands search packets with a keyword "virtual-machine" in the http request URL and save only those matched packets in the new files. We use this keyword because we know our test is about attaching/detaching VMI - "virtual" machine interface, and that will invoke REST API calls with the following http request URL:

  • http://127.0.0.1:9100/virtual-machine-interface
  • http://127.0.0.1:9100/virtual-machine

To have a quick look at the search results, use the same tshark command, but without '-w' option to print the summaries of each matching packet:

    $ tshark -r any9100.1.pcap -nY "http.request.full_uri contains virtual-machine" -t ud
     1357 2018-11-20 20:19:06.399969    127.0.0.1 → 127.0.0.1    HTTP 597 PUT /virtual-machine-interface/06bbde4f-da73-4a60-9b23-0618a0f01a6d HTTP/1.1  (application/json)
     1360 2018-11-20 20:19:06.412865    127.0.0.1 → 127.0.0.1    HTTP 482 DELETE /virtual-machine/4663093b-1a3e-4a34-868b-f4d66f59a573 HTTP/1.1
     1362 2018-11-20 20:19:06.434479    127.0.0.1 → 127.0.0.1    HTTP 615 PUT /virtual-machine-interface/06bbde4f-da73-4a60-9b23-0618a0f01a6d HTTP/1.1  (application/json)

    $ tshark -r any9100.3.pcap -nY "http.request.full_uri contains virtual-machine" -t ud
      301 2018-11-20 20:18:41.141206    127.0.0.1 → 127.0.0.1    HTTP 676 POST /virtual-machines HTTP/1.1  (application/json)
      304 2018-11-20 20:18:41.151949    127.0.0.1 → 127.0.0.1    HTTP 755 PUT /virtual-machine-interface/06bbde4f-da73-4a60-9b23-0618a0f01a6d HTTP/1.1  (application/json)

With these http packets located, we'll take a further look at the "content" within them. One good tool to use for that is "tcpflow". Below is a quick introduction of the tool from its manual:

"tcpflow is a program that captures data transmitted as part of TCP connections (flows), and stores the data in a way that is convenient for protocol analysis or debugging.  Rather than showing packet-by-packet information, tcpflow reconstructs the actual data streams and stores each flow in a separate file for later analysis."

In order to use it to extract message contents from the REST API http body, first create a folder, then run tcpflow to read the 2 files (-r) generated earlier using tshark filters. The -o option specifies the folder that the result will be saved to.

    root@cont103:~# mkdir temp2
    root@cont103:~# tcpflow -r any9100.1.vmi.pcap -o temp2
    root@cont103:~# tcpflow -r any9100.3.vmi.pcap -o temp2
    root@cont103:~# cd temp2/
    root@cont103:~/temp2# ls -lt
    total 12
    -rw-r--r-- 1 root root 3111 Dec 27 16:41 report.xml
    -rw-r--r-- 1 root root    0 Dec 27 16:41 alerts.txt
    -rw-r--r-- 1 root root 1490 Nov 20 12:19 127.000.000.001.45540-127.000.000.001.09100
    -rw-r--r-- 1 root root 1295 Nov 20 12:18 127.000.000.001.39614-127.000.000.001.09100

Two text files are generated. Each contains the REST API message content extracted from the http message bodies.

    root@cont103:~/temp2# cat 127.000.000.001.*

    POST /virtual-machines HTTP/1.1
    Host: 127.0.0.1:9100
    Content-Length: 230
    Accept-Encoding: gzip, deflate, compress
    X-Tenant-Name: admin
    Accept: */*
    X-AUTH-TOKEN: a9027b2b95034fccbfddd518781e576d
    X-Contrail-Useragent: cont103:/usr/bin/contrail-api
    User-Agent: python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-85-generic
    Content-type: application/json; charset="UTF-8"

    {
        "virtual-machine": {
            "fq_name": [
                "4663093b-1a3e-4a34-868b-f4d66f59a573"
            ],
            "perms2": {
                "global_access": 0,
                "owner": "ede15b5d21eb4401bd2a56e8d373aa71",
                "owner_access": 7,
                "share": []
            },
            "uuid": "4663093b-1a3e-4a34-868b-f4d66f59a573"
        }
    }

    PUT /virtual-machine-interface/06bbde4f-da73-4a60-9b23-0618a0f01a6d HTTP/1.1
    Host: 127.0.0.1:9100
    Content-Length: 264
    Accept-Encoding: gzip, deflate, compress
    X-Tenant-Name: admin
    Accept: */*
    X-AUTH-TOKEN: a9027b2b95034fccbfddd518781e576d
    X-Contrail-Useragent: cont103:/usr/bin/contrail-api
    User-Agent: python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-85-generic
    Content-type: application/json; charset="UTF-8"

    {   #<----- interface-attach
        "virtual-machine-interface": {
            "uuid": "06bbde4f-da73-4a60-9b23-0618a0f01a6d",
            "virtual_machine_interface_device_owner": "compute:vComp106",
            "virtual_machine_refs": [
                {
                    "to": [
                        "4663093b-1a3e-4a34-868b-f4d66f59a573"
                    ],
                    "uuid": "4663093b-1a3e-4a34-868b-f4d66f59a573"
                }
            ]
        }
    }

    PUT /virtual-machine-interface/06bbde4f-da73-4a60-9b23-0618a0f01a6d HTTP/1.1
    Host: 127.0.0.1:9100
    Content-Length: 106
    Accept-Encoding: gzip, deflate, compress
    X-Tenant-Name: admin
    Accept: */*
    X-AUTH-TOKEN: a3c115f45a2a478b9a2ebe2a082911f9
    X-Contrail-Useragent: cont101:/usr/bin/contrail-api
    User-Agent: python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-85-generic
    Content-type: application/json; charset="UTF-8"

    {   #<-----interface-detach
        "virtual-machine-interface": {
            "uuid": "06bbde4f-da73-4a60-9b23-0618a0f01a6d",
            "virtual_machine_refs": []
        }
    }

    DELETE /virtual-machine/4663093b-1a3e-4a34-868b-f4d66f59a573 HTTP/1.1
    Host: 127.0.0.1:9100
    Content-Length: 0
    Accept-Encoding: gzip, deflate, compress
    X-Tenant-Name: admin
    Accept: */*
    X-AUTH-TOKEN: a3c115f45a2a478b9a2ebe2a082911f9
    X-Contrail-Useragent: cont101:/usr/bin/contrail-api
    User-Agent: python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-85-generic
    Content-type: application/json; charset="UTF-8"

    PUT /virtual-machine-interface/06bbde4f-da73-4a60-9b23-0618a0f01a6d HTTP/1.1
    Host: 127.0.0.1:9100
    Content-Length: 124
    Accept-Encoding: gzip, deflate, compress
    X-Tenant-Name: admin
    Accept: */*
    X-AUTH-TOKEN: a3c115f45a2a478b9a2ebe2a082911f9
    X-Contrail-Useragent: cont101:/usr/bin/contrail-api
    User-Agent: python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-85-generic
    Content-type: application/json; charset="UTF-8"

    {
        "virtual-machine-interface": {
            "uuid": "06bbde4f-da73-4a60-9b23-0618a0f01a6d",
            "virtual_machine_interface_device_owner": ""
        }
    }
Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search