Support Support Downloads Knowledge Base Service Request Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Contrail] Channel error: 406: AMQP_QUEUE_DELETE_METHOD caused: PRECONDITION_FAILED - queue 'X' in vhost '/' not empty

0

0

Article ID: KB35215 KB Last Updated: 07 Dec 2019Version: 3.0
Summary:

This article explains the issue seen with respect to rabbitmq queues in Contrail version 1910 as well as in Contrail cloud 13.2 (which is also using release 1910).

Symptoms:
== Contrail control ==
control: initializing (Number of connections:2, Expected:3 Missing: Database:Cassandra, IFMap Server End-Of-RIB not computed, No BGP configuration for self)
nodemgr: active
named: active
dns: active


== Contrail config-database ==
nodemgr: active
zookeeper: active
rabbitmq: active
cassandra: active


== Contrail config ==
svc-monitor: backup
nodemgr: active
device-manager: backup
api: active
schema: backup


== Contrail webui ==
web: active
job: active


== Contrail analytics ==
nodemgr: active
api: active
collector: initializing (Number of connections:6, Expected:7 Missing: Database:Cassandra)


== Contrail analytics-alarm ==
nodemgr: active
kafka: active
alarm-gen: active


== Contrail analytics-snmp ==
snmp-collector: active
nodemgr: active
topology: active
Cause:

RabbitMq client tries to delete the 'old' queue at restart, but only if it is not empty and not in use. In some cases, the queue was not empty because it does not get deleted. Earlier, with exclusive queue flag, it can't be used further because it was locked by previous connection. After changing the flag to non-exclusive, DeclareQueue function gives an error if flags are changed on the queue. This happens in the case where some config gets pushed in the queue during restart of control node.

With 1911 fix - https://review.opencontrail.org/c/Juniper/contrail-common/+/54877, the code will "always" delete old queue even if there are messages sitting in it. With this change, old queue will always get deleted.

Solution:

How to verify this issue

Check container errorlogs - control_control_1 (contrail controller) and analytics_collector_1 (contrail collector) and find matching matching as below:

Logs can be checked using 'docker inspect control_control_1 | grep log' or 'docker logs control_control_1'

control_control_1.log indicates below:

RabbitMQ SM: Caught exception while connecting to RabbitMQ: <ip>:5673 : channel error: 406: AMQP_QUEUE_DELETE_METHOD caused: PRECONDITION_FAILED - queue 'contrail-control.overcloudo4z-cc-1.tenant.cc13-testbed-1.juniper.net' in vhost '/' not empty
RabbitMQ SM: Caught exception while connecting to RabbitMQ: <ip>:5673 : channel error: 406: AMQP_QUEUE_DELETE_METHOD caused: PRECONDITION_FAILED - queue 'contrail-control.overcloudo4z-cc-1.tenant.cc13-testbed-1.juniper.net' in vhost '/' not empty
Logs can be checked using 'docker inspect analytics_collector_1 | grep log' or 'docker logs analytics_collector_1'

analytics_collector.log indicates below:

RabbitMQ SM: Caught exception while connecting to RabbitMQ: <ip>:5673 : channel error: 406: AMQP_QUEUE_DELETE_METHOD caused: PRECONDITION_FAILED - queue 'contrail-collector.overcloudo4z-ca-1.cc13-testbed-1.juniper.net' in vhost '/' not empty
RabbitMQ SM: Caught exception while connecting to RabbitMQ: <ip>:5673 : channel error: 406: AMQP_QUEUE_DELETE_METHOD caused: PRECONDITION_FAILED - queue 'contrail-collector.overcloudo4z-ca-1.cc13-testbed-1.juniper.net' in vhost '/' not empty

Permanently, this issue will be fixed in Contrail release 1911.

As workaround, the following can be implemented:

Note: Stop control_control_1 container and analytics_collector_1 container from all nodes using command 'docker stop <container name>' before implementing the workaround.

docker exec -it contrail_config_rabbitmq bash (Note: Container name may vary as per environment eg config_database_rabbitmq_1) 

Purge those queues which are seen in control_control_1.log as well as analytics_collector.log manually using the command, 'rabbitmqctl purge_queue <queue name>'

Snippet of purging collector queue:
rabbitmqctl list_queues | grep collector
rabbitmqctl purge_queue contrail-collector.overcloudo4z-ca-0.cc13-testbed-1.juniper.net

Repeat rabbitmqctl purge for all the queues seen in control_control_1.log and analytics_collector.log

Restart contrail-controller and analytics_collector_1 using the command, 'docker restart <container name>'

docker restart control_control_1
docker restart analytics_collector_1

Note: Container names may vary as per environment. It is recommended to use docker ps and find those container names as per individual environment.
Modification History:
2019-12-07: Added note to stop containers before implementing workaround.
Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Security Alerts and Vulnerabilities

Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search