Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

CSO 5.1.1 - HA : Infrastructure Components are Unhealthy: ['Rabbitmq']

0

0

Article ID: KB35551 KB Last Updated: 06 Mar 2020Version: 1.0
Summary:
This KB article explains rabbitmq issues arriving post reboot of bare metal servers in CSO HA installations
Symptoms:
Post server reboots, Component health script sometimes may indicates rabbitmq as unhealthy(highlighted in red below)

root@startupserver1:/Contrail_Service_Orchestration_5.1.1# ./components_health.sh
INFO     Updating the mine and syncing the grains
INFO     ************************************************************************
INFO     HEALTH CHECK FOR INFRASTRUCTURE COMPONENTS STARTED IN CENTRAL ENVIRONMENT
INFO     ************************************************************************

INFO     Health Check for Infrastructure Component Saltstack Started
INFO     The Infrastructure Component Saltstack is Healthy

INFO     Health Check for Infrastructure Component Cassandra Started
INFO     The Infrastructure Component Cassandra is Healthy

INFO     Health Check for Infrastructure Component Mariadb Started
INFO     The Infrastructure Component Mariadb is Healthy

INFO     Health Check for Infrastructure Component Swift Started
INFO     Attempt: 1 - Retrying Health Check for Component Swift
INFO     The Infrastructure Component Swift is Healthy

INFO     Health Check for Infrastructure Component Redis Started
INFO     The Infrastructure Component Redis is Healthy

INFO     Health Check for Infrastructure Component Arangodb Started
INFO     The Infrastructure Component Arangodb is Healthy

INFO     Health Check for Infrastructure Component Keystone Started
INFO     The Infrastructure Component Keystone is Healthy

INFO     Health Check for Infrastructure Component Elasticsearch Started
INFO     The Infrastructure Component Elasticsearch is Healthy

INFO     Health Check for Infrastructure Component Elk_Elasticsearch Started
INFO     The Infrastructure Component Elk_Elasticsearch is Healthy

INFO     Health Check for Infrastructure Component Icinga Started
INFO     The Infrastructure Component Icinga is Healthy

INFO     Health Check for Infrastructure Component Rabbitmq Started
INFO     Attempt: 1 - Retrying Health Check for Component Rabbitmq
INFO     Attempt: 2 - Retrying Health Check for Component Rabbitmq
ERROR    The Infra Component : Rabbitmq is Unhealthy

INFO     Health Check for Infrastructure Component Etcd Started
INFO     The Infrastructure Component Etcd is Healthy

INFO     Health Check for Infrastructure Component Rsyslog Started
INFO     The Infrastructure Component Rsyslog is Healthy

INFO     Health Check for Infrastructure Component Kubernetes Started
INFO     The Infrastructure Component Kubernetes is Healthy

INFO     Health Check for Infrastructure Component Elk_Logstash Started
INFO     The Infrastructure Component Elk_Logstash is Healthy

INFO     Health Check for Infrastructure Component Elk_Kibana Started
INFO     The Infrastructure Component Elk_Kibana is Healthy

INFO     Health Check for Infrastructure Component Zookeeper Started
INFO     The Infrastructure Component Zookeeper is Healthy

INFO     Health Check for Infrastructure Component Contrail_Analytics Started
INFO     The Infrastructure Component Contrail_Analytics is Healthy

INFO     Overall result:
INFO             The following Infrastructure Components are Healthy:
INFO                     ['Saltstack', 'Cassandra', 'Mariadb', 'Swift', 'Redis', 'Arangodb', 'Keystone', 'Elasticsearch', 'Elk_Elasticsearch', 'Icinga', 'Etcd', 'Rsyslog', 'Kubernetes', 'Elk_Logstash', 'Elk_Kibana', 'Zookeeper', 'Contrail_Analytics']
ERROR    The following Infrastructure Components are Unhealthy: ['Rabbitmq']
Please Check the logs:
===========================
Info Logs Path: /Contrail_Service_Orchestration_5.1.1/logs/health_check.log
Cause:
The cause of this issue is rabbitmq cluster may not be formed correctly post reboot.

Logging into rabbitmq webui using : http://<>:15672 , (username: cspmq, password can be obtained from command ./python.sh deploy_manager/utils/decrypt_password.py )
issue can be verified by clicking on overview tab below. As we can see, one of the rabbitmq node in the HA cluster is not running, which triggers the "unhealthy" message in health check.

Solution:
Rabbitmq can be recovered by running recovery script, which is located in the install directory of Contrail_Service_Orchestration_5.1.1

root@startupserver1:/Contrail_Service_Orchestration_5.1.1# ./recovery.sh
***************
This tool assists you recover your CSO setup.
***************
 
 Following components can be recovered
 
1: mariadb
2: vrr
3: icinga
4: rabbitmq
 
Specify one of the component to recover (In Number) : 4
INFO     Started recovering rabbitmq component at 2020-03-04 21:31:02.537449 ...
INFO     Rabbitmq failure recovery is initiated, it will take couple of mintues...
INFO     Rabbitmq recovery started. Please wait...
INFO     Completed recovering rabbitmq component at 2020-03-04 21:48:11.882178 .
INFO     Time taken to recover 0:17:09.344729

Verification can be done by running components_health.sh script again after recover step taken.

Note : In case of rabbitmq failure, error is printed as "INFO   Unable to Recover Rabbitmq" , this will be enhanced in future and it will print failure as ERROR instead of INFO
Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search