Support Support Downloads Knowledge Base Apex Support Portal Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Contrail] How to apply hotfix for Contrail processes with no internet access in undercloud/overcloud nodes



Article ID: KB37293 KB Last Updated: 27 Aug 2021Version: 1.0

This article details how to apply a hotfix that is provided to a customer for Contrail processes when there is no Internet access in the undercloud and overcloud nodes.

Note that this hotfix can be for any Contrail process such as contrail-vrouter-agent and contrail-controller-control-control.

Some important points that should be considered before implementing this procedure are as follows:

  • This procedure is applicable when the undercloud and overcloud nodes do not have Internet connectivity. If there is Internet connectivity, the image can be downloaded directly from

  • The primary difference between this procedure and the one performed when there is Internet connectivity is the way in which the image becomes accessible to the compute/controller that needs to be patched with the hotfix.

  • During this procedure, you should always use "docker load" instead of "docker import." When you use "docker import," all the labels and other parameters attached to the image are dropped. This in-turn generates an error while executing the final steps of the procedure.

  • Updating overcloud_images.yaml is important because it is used for stack update as well. Performing a stack update with unmodified overcloud_images.yaml will result in rollback of the hotfix that you applied.  


To apply the hotfix for Contrail processes, perform the following:

  1. Get contrail-<Process_Container_Name>.<Version> from by using the pull command.

. . . . .
Login Succeeded
root@mgmt-host:~# docker pull<Folder_Location>/contrail-<Process_Container_Name>:<Version>
<Version>: Pulling from <Folder_Location>/contrail-<Process_Container_Name>
c9281c141a1b: Pull complete
. . . . .
9b1ba50da51d: Pull complete
Digest: sha256:171563f3bdac1142071f0fa0c780e89e14fa59eb3cdc051076ffbb1ac955ae5a
Status: Downloaded newer image for<Folder_Location>/contrail-<Process_Container_Name>:<Version><Folder_Location>/contrail-<Process_Container_Name>:
  1. Get the docker image ID.

root@mgmt-host:~#  docker image ls | grep -e REPO -e vrouter
REPOSITORY                                           TAG              IMAGE ID       CREATED         SIZE<Folder_Location>/contrail-<Process_Container_Name>   <Version>   79f0c703a5de   2 weeks ago     2.49GB
  1. Save the docker image to a tar file for transfer to the required nodes.

root@mgmt-host:~# docker image save -o <Tar_File_Save_Location>/contrail-<Process_Container_Name>_<Version>.tar 79f0c703a5de
  1. Compress the tar file with gzip.

root@mgmt-host:~# ls -cltr <Tar_File_Save_Location>/contrail-<Process_Container_Name>*.tar
-rw------- 1 root  root  2515657728 Jun 22 15:53 contrail-<Process_Container_Name>_<Version>.tar
root@mgmt-host:~# gzip <Tar_File_Save_Location>/contrail-<Process_Container_Name>_<Version>.tar
root@mgmt-host:~# ls -cltr <Tar_File_Save_Location>/contrail-<Process_Container_Name>*.gz
-rw------- 1 root root 787442747 Jun 22 15:56 contrail-<Process_Container_Name>_<Version>.tar.gz
  1. Transfer the compressed image archive to <Dir_Tar_File_Save_Location> in the RedHat OpenStack Director node and extract the .tar file.

[stack@XYZ ]$ gunzip <Dir_Tar_File_Save_Location>/contrail-<Process_Container_Name>_<Version>.tar.gz
  1. Confirm that docker-distribution.service is running on the RHOSP Director node.

[stack@XYZ ~]$ systemctl status docker-distribution.service
● docker-distribution.service - v2 Registry server for Docker
   Loaded: loaded (/usr/lib/systemd/system/docker-distribution.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2020-02-21 12:02:17 AEDT; 1 years 4 months ago
 Main PID: 4591 (registry)
    Tasks: 28
   CGroup: /system.slice/docker-distribution.service
           └─4591 /usr/bin/registry serve /etc/docker-distribution/registry/config.yml
  1. Fetch the IP address details of the repository and registry that are listening on port 8787. 

[stack@XYZ ~]$ sudo ss -tlnp | grep registry
LISTEN     0      128    <Local_Dir_Repo_IP>:8787                     *:*                   users:(("registry",pid=4591,fd=3))
[stack@XYZ ~]$ sudo ss -tnlp | grep registry
LISTEN     0      128    <Local_Repo_IP>:8787                     *:*                   users:(("registry",pid=25785,fd=3))

The IP address details will be used with the local repositories on the overcloud nodes. 

  1. Test whether the docker image catalog is available from the registry:

[stack@XYZ]$ curl -X GET http://Local_Dir_Repo_IP:8787>/v2/_catalog | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4901    0  4901    0     0  91203      0 --:--:-- --:--:-- --:--:-- 94250
    "repositories": [
. . . . .
  1. Load the contents of the tarball to create a filesystem docker image.

[stack@XYZ ]$ sudo docker image load -i <Dir_Tar_File_Save_Location>/contrail-<Process_Container_Name>_<Version>.tar
6ad80b248b34: Loading layer [==================================================>] 2.048 kB/2.048 kB
. . . . .
1e9e613db6aa: Loading layer [==================================================>] 14.85 k
[stack@XYZ ]$ sudo docker image ls | grep -e REPO -e none
REPOSITORY                                                                                             TAG                      IMAGE ID            CREATED             SIZE
<none>                                                                                                 <none>                   79f0c703a5de        2 weeks ago         2.49 GB
  1. Tag the container with correct information, and then upload to the local repository of the node for local registry access.

[stack@XYZ ]$ sudo docker tag 79f0c703a5de <Local_Repo_IP>:8787/<New_Location>/contrail-<Process_Container_Name>:<Version>
[stack@XYZ ]$ sudo docker image ls | grep -e REPO -e <New_Location>
REPOSITORY                                                                                             TAG                      IMAGE ID            CREATED             SIZE
<Local_Repo_IP>:8787/<New_Location>/contrail-<Process_Container_Name>                                      <Version>           79f0c703a5de        2 weeks ago         2.49 GB
[stack@XYZ ]$ sudo docker push <Local_Repo_IP>:8787/<New_Location>/contrail-<Process_Container_Name>:<Version>
The push refers to a repository [<Local_Repo_IP>:8787/<New_Location>/contrail-<Process_Container_Name>]
1e9e613db6aa: Pushed
. . . . .
<Version>: digest: sha256:171563f3bdac1142071f0fa0c780e89e14fa59eb3cdc051076ffbb1ac955ae5a size: 3252
  1. Modify /home/stack/overcloud/deployment/overcloud_images.yaml. This file is used for initial deployment as well a stack update. So if you miss this step, then the next stack update will roll back the hotfix that you applied.

[stack@XYZ ]$ cd /home/stack/overcloud/deployment/
[stack@XYZ deployment]$ cp overcloud_images.yaml overcloud_images.yaml_pre-vrouterHF
Modify line for DockerContrailVrouterAgentContainerName and DockerContrailVrouterAgentImageName for persistance.
[stack@XYZ deployment]$ grep <New_Location> overcloud_images.yaml
  DockerContrailVrouterAgentContainerName: <Local_Repo_IP>:8787/<New_Location>/contrail-<Process_Container_Name>:<Version>
  DockerContrailVrouterAgentImageName: <Local_Repo_IP>:8787/<New_Location>/contrail-<Process_Container_Name>:<Version>
[stack@XYZ deployment]$ diff overcloud_images.yaml overcloud_images.yaml_pre-vrouterHF
<   DockerContrailVrouterAgentContainerName: <Local_Repo_IP>:8787/<New_Location>/contrail-<Process_Container_Name>:<Version>
>   DockerContrailVrouterAgentContainerName: <Local_Repo_IP>:8787/<Old_Location>/contrail-<Process_Container_Name>:1908.47-rhel
<   DockerContrailVrouterAgentImageName: <Local_Repo_IP>:8787/<New_Location>/contrail-<Process_Container_Name>:<Version>
>   DockerContrailVrouterAgentImageName: <Local_Repo_IP>:8787/<Old_Location>/contrail-<Process_Container_Name>:1908.47-rhel
  1. Log in to the hypervisor that you wish to update with the hotfix image container. Capture the output of the following commands and save a copy on Notepad.

[stack@XYZ ~]$ ssh heat-admin@ABC-Compute.Domain
[heat-admin@ABC-Compute ~]$ sudo su -
[root@ABC-Compute ~]# docker image ls | grep -i <Process_Container_Name>
REPOSITORY                                                           TAG                        IMAGE ID            CREATED             SIZE
<Local_Repo_IP>:8787/<Old_Location>/contrail-<Process_Container_Name>                 1908.47-rhel               af0b73228dba        22 months ago       3.2 GB
[root@ABC-Compute ~]# docker ps -a | grep -i <Process_Container_Name>
CONTAINER ID        IMAGE                                                                                   COMMAND                  CREATED             STATUS                    PORTS               NAMES
3040418ab8ad        <Local_Repo_IP>:8787/<Old_Location>/contrail-<Process_Container_Name>:1908.47-rhel           "/ /u..."   9 months ago        Up 9 months  contrail-<Process_Container_Name>
[root@ABC-Compute ~]# curl -X GET http://<Local_Repo_IP>:8787/v2/_catalog | python -m json.tool | grep grep -i <Process_Container_Name>
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4905    0  4905    0     0   117k      0 --:--:-- --:--:-- --:--:--  122k
  1. Back up /var/lib/tripleo-config/hashed-docker-container-startup-config-step_5.json.

[root@ABC-Compute ~]# cd /var/lib/tripleo-config/
[root@ABC-Compute tripleo-config]# ls -al hashed-docker-container-startup-config-step_5.json
-rw-------. 1 root root 4223 Sep 23  2020 hashed-docker-container-startup-config-step_5.json
[root@ABC-Compute tripleo-config]# cp hashed-docker-container-startup-config-step_5.json hashed-docker-container-startup-config-step_5.json_pre-vrouterHF
[root@ABC-Compute tripleo-config]# diff hashed-docker-container-startup-config-step_5.json hashed-docker-container-startup-config-step_5.json_pre-vrouterHF
  1. Edit the new tag as shown in the following file, which corresponds to the contrail-<Process_Container_Name> container in /var/lib/tripleo-config/hashed-docker-container-startup-config-step_5.json.

[root@ABC-Compute tripleo-config]# diff hashed-docker-container-startup-config-step_5.json hashed-docker-container-startup-config-step_5.json_pre-vrouterHF
<     "image": "<Local_Repo_IP>:8787/<New_Location>/contrail-<Process_Container_Name>:<Version>",
>     "image": "<Local_Repo_IP>:8787/<Old_Location>/contrail-<Process_Container_Name>:<Old_Version>",
  1. Deploy the new contrail-<Process_Container_Name> container.

[root@ABC-Compute tripleo-config]# step=5
[root@ABC-Compute tripleo-config]# paunch --debug apply --managed-by paunch --config-id tripleo_step$step --file /var/lib/tripleo-config/hashed-docker-container-startup-config-step_$step.json
stdout: Trying to pull repository <Local_Repo_IP>:8787/<New_Location>/contrail-<Process_Container_Name> ...
<Version>: Pulling from <Local_Repo_IP>:8787/<New_Location>/contrail-<Process_Container_Name>
c9281c141a1b: Already exists
31114e120ca0: Already exists
7ad41b0756da: Pulling fs layer
. . . . .
78c3993e692f: Pull complete
9b1ba50da51d: Pull complete
Digest: sha256:171563f3bdac1142071f0fa0c780e89e14fa59eb3cdc051076ffbb1ac955ae5a
Status: Downloaded newer image for <Local_Repo_IP>:8787/<New_Location>/contrail-<Process_Container_Name>:<Version>

stdout: dc1557ec8a48970650e93b141593ef6d9b75a34a3b90d7466ba97bed570cf259

  1. Confirm that the new vRouter container is loaded and that contrail-status is functional.

[root@ABC-Compute tripleo-config]# docker ps -a | grep <New_Location>
dc1557ec8a48        <Local_Repo_IP>:8787/<New_Location>/contrail-<Process_Container_Name>:<Version>        "/ /u..."   About a minute ago   Up About a minute                             <Process_Container_Name>
[root@ABC-Compute tripleo-config]# contrail-status
Pod      Service     Original Name                State    Id            Status
vrouter  agent       contrail-<Process_Container_Name>       running  dc1557ec8a48  Up About a minute
vrouter  agent-dpdk  contrail-<Process_Container_Name>-dpdk  running  d2bbc940c6eb  Up 9 months
vrouter  nodemgr     contrail-nodemgr             running  f2b22c17248a  Up 9 months

vrouter DPDK module is PRESENT
== Contrail vrouter ==
nodemgr: active
agent: active
Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search