Support Support Downloads Knowledge Base Juniper Support Portal Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Contrail] Understanding RabbitMQ ghost queue behavior after changing hostname

0

0

Article ID: KB37116 KB Last Updated: 18 Jun 2021Version: 1.0
Summary:
RabbitMQ queue names are built from the hostname. Hence, changing the hostname can affect the RabbitMQ cluster. This article explains what to check in RabbitMQ cluster after such changes.
Symptoms:

Contrail RabbitMQ subscriber queue names are created from the subscriber hostname and bound to vnc_config exchange to get messages published by contrail-api. Relevant subscribers' queue is deleted when subscribers' service is started while connecting to the RabbitMQ. If the hostname is changed, the relevant queue name is changed and will not match the previous queue name. Therefore, old queues will not be deleted. They will continue to get messages as it is still bound to the exchange. As result, the number of messages will grow. Over time time, it can result in slow messages propagating from the publisher to the subscriber.

RabbitMQ subscribers are all other config services, control node, and some analytics services. One of the symptoms can be missing system route target for a routing instance when a new virtual network is created, or control node missing new updates which can be reflected in missing networking connectivity for workloads in virtual networks created during the issue time.

Cause:

A known behavior in 4.x, 19xx and 20xx releases is that we can end up having ghost queues in some corner cases caused by /etc/hosts changes. 

A queue name is obtained from /etc/hosts file by the following code instructions when a service is started and while trying to connect to RabbitMQ:

q_name = 'vnc_config.%s-%s' % (socket.getfqdn(host_ip), listen_port)
>>> import socket
>>> socket.getfqdn("10.0.0.14")
'overcloud-contrailcontroller-2.tenant.localdomain'

In a corner case, when the /etc/hosts file is changed by modifying or adding another domain name, the system can end up with a new queue after restarting the service. That can happen if a new domain name is changed or inserted before the existing one (for the same IP).

Subscriber, the relevant restarted service, will be listening to the “New Queue”. The “Old Queue” will not be deleted after the service restarts and it will not have any subscribers. On the other side, it is still bound to the vnc_config exchange and will continue getting messages from the exchange.

Queues can be listed by using rabbitmqctl list_queues command or by using RabbitMQ Management Plugin. Here is an example with resulted queues after changing the name to fully qualified domain name:

icmlw-p1-jcontvm-001a:contrail-snmp-collector:0.icmlw-p1-jcontvm-001a 284141
icmlw-p1-jcontvm-001b:contrail-snmp-collector:0.icmlw-p1-jcontvm-001b 284143
icmlw-p1-jcontvm-001c:contrail-snmp-collector:0.icmlw-p1-jcontvm-001c 284147
icmlw-p1-jcontvm-001a.itpc.uk.pri.o2.com:contrail-snmp-collector:0.icmlw-p1-jcontvm-001a.itpc.uk.pri.o2.com 0
icmlw-p1-jcontvm-001b.itpc.uk.pri.o2.com:contrail-snmp-collector:0.icmlw-p1-jcontvm-001b.itpc.uk.pri.o2.com 0
icmlw-p1-jcontvm-001c.itpc.uk.pri.o2.com:contrail-snmp-collector:0.icmlw-p1-jcontvm-001c.itpc.uk.pri.o2.com 0
Solution:

This old queue must be deleted manually to prevent large numbers of messages building up and affecting the RabbitMQ cluster.

In the example above, all old queues 'icmlw-p1-jcontvm-001x:contrail-snmp-collector:0.icmlw-p1-jcontvm-001x' must be manually deleted. The RabbitMQ queue can be deleted either by using rabbitmqctl delete_queue command or from a RabbitMQ Management Plugin.

Related Links

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search