Support Support Downloads Knowledge Base Apex Support Portal Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Contrail] How to delete stale/orphaned Virtual Interfaces from Contrail 1908 using "vrouter-port-control"

0

0

Article ID: KB37190 KB Last Updated: 28 Sep 2021Version: 2.0
Summary:

Sometimes, users may observe a discrepancy between the number of instances and ports/interfaces present in the OpenStack database and on the Contrail Web User Interface in a Contrail Networking (CN) with Red Hat OpenStack (RHOSP) setup. When the setup is analyzed, more number of stale port entries are seen on the Contrail WebUI attached to the stale instances than in the OpenStack database.

This article describes how to delete the stale/orphaned interfaces from the Contrail WebUI and align the entries with the entries in the OpenStack database.

Symptoms:

A mismatch is seen between the number of instances and ports/interfaces present in the OpenStack database and on the Contrail Web UI. When the setup is analyzed, more number of stale port entries are seen on the Contrail WebUI attached to the stale instances than in the OpenStack database.

The mismatch is demonstrated below:

The number of instances and ports/interfaces in OpenStack are as below:

(Overcloud) [stack@Openstack ~]$ openstack server list --all-projects -c ID -f value | wc -l
62                                                                     >>>> No of instances
(Overcloud) [stack@Openstack  ~]$ openstack port list -c ID -f value | wc -l
201                                                                    >>>> No of ports/interfaces
(Overcloud) [stack@Openstack  ~]$

The number of instances and ports/interfaces on the Contrail WebUI are as below:

616 instances 
1247 Ports/interfaces

From the compute node, many interfaces with Vrf:65535 (orphaned ViFs) are seen as shown below:

[root@XYZ-Compute ~]# vif --list |grep -A2 "vif0/"       >>>> Returns the list of all interfaces present on the compute node with the name containing "vif0/"
--
vif0/243    PMD: tap40e1991b-bf
Type:Virtual HWaddr:XX:XX:YY:XX:ZZ:XX IPaddr:A.B.C.D
Vrf:65535 Mcast Vrf:65535 Flags:L3L2DEr QOS:-1 Ref:19      >>>> Vrf pointing to 65535
--
[root@XYZ-Compute ~]#
Solution:

Important Note: Identification of UUIDs is required for all the stale entries present on the Contrail WebUI, but not present in OpenStack. Deleting a port/interface that is in use by an active/stopped instance could lead to the instance being in bad state. So UUID identification is really important here.

Below are the steps for completing this procedure:

  1. Gathering Information

  2. Identifying Stale Ports/Interface UUIDs

  3. Deleting Identified Stale Entries Using the vrouter-port-control Command

Gathering Information

The following data must be collected to identify the stale entries that need to be deleted:

Generic Data from Overall Setup

  • Detailed port and instance information present in OpenStack 

  • Detailed port and instance information present on Contrail WebUI

Specific Data from Impacted Computes

  • Introspects

In case of HTTPS (SSL Enabled)

curl --ssl -k --key /etc/contrail/ssl/private/server-privkey.pem --cert /etc/contrail/ssl/certs/server.pem https://<Compute-IP>:8085/Snh_KInterfaceReq?if_id=
curl --ssl -k --key /etc/contrail/ssl/private/server-privkey.pem --cert /etc/contrail/ssl/certs/server.pem https://<Compute-IP>:8085/Snh_KSyncItfReq?
curl --ssl -k --key /etc/contrail/ssl/private/server-privkey.pem --cert /etc/contrail/ssl/certs/server.pem https://<Compute-IP>:8085/Snh_ItfReq?

In case of HTTP (SSL Disabled)

curl http://<Compute-IP>:8085/Snh_KInterfaceReq?if_id=
curl http://<Compute-IP>:8085/Snh_KSyncItfReq?
curl http://<Compute-IP>:8085/Snh_ItfReq?
  • Information about all VMs present on the compute node

[root@XYZ-Compute ~]# virsh list --all             >>>> Output list of all runnnig and stopped Instances on that compute
  • Archive file containing all files under /var/lib/contrail/ports/* from the compute node

/var/lib/contrail/ports/ contains files created for each port which is spawned on this compute. Each file for a port contains detailed information for that port.

Identifying Stale Ports/Interface UUIDs

Use the following steps to identify the UUIDs of stale entries:

  1. Analyze the introspects collected from the previous task and identify all the port UUID entries with error states as shown below:

<ItfSandeshData>
<index type="i32" identifier="1">122</index>
<name type="string" identifier="2">tap00a52864-f0</name>
<uuid type="string" identifier="3">00a52864-f0ee-4528-a871-791170dc0589</uuid>
<vrf_name type="string" identifier="4" link="VrfListReq">--ERROR--</vrf_name>
<active type="string" identifier="5">Inactive &lt; vn-null vrf-null os-state-down ipv4_inactive ipv6_inactive l2_inactive  &gt;</active>
<ipv4_active type="string" identifier="49">Ipv4 Inactive &lt; vn-null vrf-null os-state-down l3-disabled  &gt;</ipv4_active>
<l2_active type="string" identifier="28">L2 Inactive &lt; vn-null vrf-null os-state-down l2-disabled  &gt;</l2_active>
<ip6_active type="string" identifier="35">Ipv6 Inactive &lt; vn-null vrf-null os-state-down l3-disabled no-ipv6-addr  &gt;</ip6_active>
  1. Create a list of Identified Port UUIDs from the above steps. Match the UUID list with the port UUIDs collected from OpenStack. If there is a match, remove that UUID from the Identified Port UUID list.

  2. Now match the UUIDs of all the ports attached to all the instances running on that compute node against the Identified Port UUID list. If there is a match, remove that UUID from the Identified Port UUID list.

After completing the above steps, you will get an identified Port UUID list, which contains all the stale/orphaned interfaces present on that compute node. Now you can delete the identified ports to remove the stale entries in the next step.

Deleting Stale Entries Using the "vrouter-port-control" Command

To add or delete ports present on the compute node, you can use an existing utility script on the compute node's "Contrail-vRouter-agent" container. 

Pre-Execution check 

cd /var/lib/contrail/ports/ ; ls -l | wc -l               >>>> Count the number of port files present before deletion.

The following command can be used for deleting stale entries from the compute node:

vrouter-port-control --oper=delete --uuid=UUID     >>>> UUID can be replaced by UUIDs present in "Identified Port UUID list" from the earlier section.

Verification

cd /var/lib/contrail/ports/ ; ls -l | wc -l          >>>> Count the number of port files present after deletion; should be 1 less than the previous step.

Make sure to confirm the Contrail WebUI for this port. It should have been deleted and the corresponding instance attached to this port should also have been deleted.

Note: The above procedure is recommended when you have fewer number of stale entries. If you have several entries to delete, it might require a lot of time. In that case, you can create a command file or use the following script to help you with the task.

Python Script for Stale Entries Deletion

A Python script has been created to delete all stale entries from a compute node at the same time. This is useful when you have several stale entries on the compute node. For this script, you need to create a "Computename_UUID_Entries.txt" file that contains all the port UUID entries from the "Identified Port UUID list" that was generated in the earlier section.

Caution: The following script has not been tested in a production environment, so it is not recommended for use in a production environment. Git Link for the Script: https://css-git.juniper.net/rkwatra/stale-entries-deletion

Modification History:

2021-09-28: Git link to the script added to the Solution section

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search