Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[CSO] Red indices in Elasticsearch causing site creation failures

0

0

Article ID: KB35833 KB Last Updated: 19 Jun 2020Version: 1.0
Summary:

Elasticsearch is a NoSQL database. In Contrail Service Orchestration (CSO), the Elasticsearch cluster will have dedicated VMs deployed in HA mode. When there are red indices in the Elasticsearch database, it can affect site deletion and re-creation through the CSO GUI.

This article details the steps to detect such red shards in the Elasticsearch database and to check the overall health of the Elasticsearch database.

 

Symptoms:

When we delete a site and re-create it, we can see the following error logs in kibana, which points to a stale entry in the database. 

<Error log Snip  >

"status_code": "<status code",
"error_tag": "Refs Exist",
"error_message": "Operation can not be completed since the resource is still being referred to. "
error_diag": "This error occurs when a resource is attempted for modification or deletion when there are child resources still referring to it. ",
"error_code": "<error code>

 

Solution:

To check the health of Elasticsearch components, run the components_health.sh script.

Note

  • In CSO 5.x release, you need to run the components_health.sh script from the startup server. The script can be found in /root//Contrail_Service_Orchestration_5.x.x/.

  • In CSO 4.x release, you need to run the components_health.sh script from the installer server. The script can be found in /root//Contrail_Service_Orchestration_4.x.x/.

2019-06-06 18:38:09 INFO     __main__ check_infracomponent.py Health Check for Infrastructure Component ElasticSearch Started
2019-06-06 18:38:09 INFO     __main__ check_infracomponent.py Attempt: 1 - Retrying Health Check for Component ElasticSearch
2019-06-06 18:38:40 INFO     __main__ check_infracomponent.py Attempt: 2 - Retrying Health Check for Component ElasticSearch
2019-06-06 18:39:10 ERROR    __main__ check_infracomponent.py The Infra Component : ElasticSearch is Unhealthy

As shown in the above output, Elasticsearch is unhealthy.

 

Checking For Red Indices

To check for any red indices in the Elasticsearch database, perform the following steps:

  1. Log in to the Elasticsearch instance with the username/password provided during CSO installation.

  2. Use the following curl command and look for the red keyword in the command output.

curl -k -uadmin:< #enter elasticsearch password here#> https://<elastic search cluster ip:9200/_cat/indices?v | grep "red"
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
100 12 red open ems-central-device oXsqQDS1SR6c2CoSFeXusg 2 1 562 0 484.8kb 242.4kb 4k 100 124k 0 0 131k 0 --:--:-- --:--:-- --:--:-- 131k

Note: In on-prem deployments, we can run the following command from the central infra VM.

curl -k -uadmin:< #enter elasticsearch password here#> https://<elastic search cluster ip:9200/_cat/indices?v | grep "red"

As seen in the above output, there are red indices in the database, which need to be deleted followed by a re-indexing of the red indices.

Note: Since a database is a critical component, performing a delete operation would require appropriate guidance. Contact Support for guidance.

 

Verification

After the red indices are deleted, check cluster health by using the following curl command. The cluster “status” should now be green.

curl -k -uadmin:<#enter elasticsearch password here#> https://<elastic search cluster ip >:9200/_cluster/health?pretty

{
  "cluster_name" : "YX38GA",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 2563,
  "active_shards" : 5127,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0

 

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search