Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Contrail] Troubleshooting with 'rabbitmq' CLI commands

0

0

Article ID: KB34424 KB Last Updated: 14 Jul 2020Version: 3.0
Summary:

Contrail uses the messaging services of rabbitmq when coordinating operations and status information among several of its services. Any inconsistencies within a rabbitmq cluster may render the different services of Contrail in a non-working state. rabbitmqctl is a command-line tool that is frequently used for managing and troubleshooting a rabbitmq cluster of nodes. It can be used to perform several diagnostic and data-collection actions by connecting to the target rabbitMQ node on a dedicated CLI communication port.

If any of the Contrail services fails due to its dependencies on rabbitmq, it is typically reflected in the output of the contrail-status command or in the Contrail logs of the corresponding service that is failing. In such cases, it is advisable to take a look at the status of the rabbitmq cluster and verify the health of the different components of rabbitmq.

This article lists some of the useful rabbitmqctl commands that you should keep handy when troubleshooting issues in Contrail that are related to rabbitmq.

 

Solution:

Some useful rabbitmqctl commands are given below:

  1. #rabbitmqctl cluster_status

This command displays all the nodes in the cluster grouped by node type, together with the currently running nodes. This command is useful in determining the overall health of the rabbitmq cluster.

If any of the nodes are down, then cluster_status output will show nodedown in the alarm section:

To print the status information for a particular node, use the rabbitmqctl status command. This command will display the status of the rabbitmq server/broker node on which the command is run:

  1. #rabbitmqctl list_queues

This is one of the rabbitmq topology introspect commands that is useful to see the list of available queues. By default, this command returns a list of queues with their corresponding name and queue depth. Queue depth indicates the sum of ready and unacknowledged messages in a queue.

  1. #rabbitmqctl list_connections

This command lists the state of TCP connections between each of the rabbitmq servers/broker nodes and the clients. The clients in this case will be different Contrail services that are using the messaging services of rabbitmq.

This is a cluster wide command, which means running this command on any nodes gets the status across the entire cluster.

Example

# rabbitmqctl list_connections
  
Listing connections
guest   172.18.101.102  53910   running
guest   172.18.101.101  48842   running
guest   172.18.101.101  48844   running
guest   172.18.101.101  48845   running
guest   172.18.101.101  48846   running
guest   172.18.101.101  48847   running
guest   172.18.101.101  48849   running
guest   172.18.101.101  48850   running
guest   172.18.101.101  48851   running
guest   172.18.101.101  48852   running
guest   172.18.101.101  48854   running
guest   172.18.101.101  48853   running
 ---------------Output Skipped------------------

In the above output, guest is the username of the rabbitmq user the client is using to connect to the server.

IP addresses are the IP addresses of the rabbitmq servers along with the TCP port numbers on the servers the clients are connecting to.

The last column of the output indicates the state of the TCP connections. The connection state could be one of the following:

 

  1. #rabbitmqctl list_bindings 

In rabbitmq, messages are not published directly to a queue. Instead, the producer sends messages to an exchange. An exchange accepts messages from the producer application and routes them to the appropriate message queues. A binding is a "link" that is set up to bind a queue to an exchange.

This command output lists all the available queues along with their linked exchanges. If an exchange does not exist for a queue, then the messages sent to that queue will not be received by the subscribers of that queue.

  1. #rabbitmqctl list_channels

A channel is a virtual connection inside a TCP connection. When a Contrail service is publishing or consuming messages from a queue, it is done over a channel. Thus, it is important at the time of troubleshooting to validate if an active channel exists between the client and the broker/server.

There can be multiple channels established over a single TCP connection.

In the above output:

  • 1st column: Is the rabbitmq PID of the process associated with the connection

  • 2nd column: Is the username associated with the channel

  • 3rd column: Is the number of consumers retrieving messages via the channel

  • 4th column: Indicates unacknowledged messages, which are the number of messages delivered via this channel but not yet acknowledged

  • Each channel can also be identified based on the TCP connection it resides in; we can view this by running the above command with the "name" option.

  1. #rabbitmqctl list_consumers

This command lists all the subscriptions to a queue´s message stream.

Each printed line is separated by tab characters and shows the following:

  • The name of the queue subscribed to

  • The PID of the channel process via which the subscription was created and is managed

  • The consumer tag which uniquely identifies the subscription within a channel

  • A boolean indicating whether acknowledgements are expected for messages delivered to this consumer

  1. #rabbitmqctl eval "rabbit_diagnostics:maybe_stuck()".

This command periodically samples stack traces of all rabbitmq PIDs ("lightweight threads" also called "Erlang Processes") on the node and reports the processes for which stack trace samples are identical.

Identical samples may indicate that the process is not making any progress. However, it is not necessarily an indication of a problem.

 

Modification History:
  • 2020-07-03: Fixed typo for command 6 (added rabbitmqctl list_consumers)

  • 2020-07-14: Fixed Step 7, added . at the end of the command

 

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search