Support Support Downloads Knowledge Base Juniper Support Portal Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Contrail] R1909 Contrail-cassandra-repair.py not invoked by Contrail nodemgr

0

0

Article ID: KB36321 KB Last Updated: 01 Feb 2021Version: 1.0
Summary:

This article details a specific scenario of Cassandra database repair, its impact, a workaround, and the fixed version information.

 

Symptoms:

The schema container is crashing in all three Contrail nodes:

[contrail-schema] [ERROR]: Cassandra connection down. Exception in<unbound method ColumnFamily.multiget>: Retried 16 times. Last failure was timeout: timed out

contrail-config-database-nodemgr.log indicates failed to get database usage: Timed out and failed to get nodetool compactionstats

 

Solution:

The issue is fixed in Contrail R1911 and later where the contrail-cassandra-repair.py script is invoked to perform periodic nodetool repair -pr from nodemgr every 24 hours by default.

Meanwhile, as a workaround, you can run the following query on all Contrail nodes:

docker exec -it config_database_cassandra_1 nodetool -p 7201 repair -pr config_db_uuid

Note: The container name may change based on deployment type.

Depending on the database size, sometimes you may be required to run nodetool with the --full argument: nodetool repair -p 7201 -pr --full.

Reference: https://review.opencontrail.org/c/Juniper/contrail-controller/+/54625

 

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search