Elasticsearch is a NoSQL database. In Contrail Service Orchestration (CSO), the Elasticsearch cluster will have dedicated VMs deployed in HA mode. When there are red indices in the Elasticsearch database, it can affect site deletion and re-creation through the CSO GUI.

This article details the steps to detect such red shards in the Elasticsearch database and to check the overall health of the Elasticsearch database.

 

When we delete a site and re-create it, we can see the following error logs in kibana, which points to a stale entry in the database. 

<Error log Snip  >

"status_code": "<status code",
"error_tag": "Refs Exist",
"error_message": "Operation can not be completed since the resource is still being referred to. "
error_diag": "This error occurs when a resource is attempted for modification or deletion when there are child resources still referring to it. ",
"error_code": "<error code>

 

To check the health of Elasticsearch components, run the components_health.sh script.

Note

• In CSO 5.x release, you need to run the components_health.sh script from the startup server. The script can be found in /root//Contrail_Service_Orchestration_5.x.x/.

• In CSO 4.x release, you need to run the components_health.sh script from the installer server. The script can be found in /root//Contrail_Service_Orchestration_4.x.x/.

2019-06-06 18:38:09 INFO     __main__ check_infracomponent.py Health Check for Infrastructure Component ElasticSearch Started
2019-06-06 18:38:09 INFO     __main__ check_infracomponent.py Attempt: 1 - Retrying Health Check for Component ElasticSearch
2019-06-06 18:38:40 INFO     __main__ check_infracomponent.py Attempt: 2 - Retrying Health Check for Component ElasticSearch
2019-06-06 18:39:10 ERROR    __main__ check_infracomponent.py The Infra Component : ElasticSearch is Unhealthy

As shown in the above output, Elasticsearch is unhealthy.

 

Checking For Red Indices

To check for any red indices in the Elasticsearch database, perform the following steps:

1. Log in to the Elasticsearch instance with the username/password provided during CSO installation.

2. Use the following curl command and look for the red keyword in the command output.

curl -k -uadmin:< #enter elasticsearch password here#> https://<elastic search cluster ip:9200/_cat/indices?v | grep "red"

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
100 12 red open ems-central-device oXsqQDS1SR6c2CoSFeXusg 2 1 562 0 484.8kb 242.4kb 4k 100 124k 0 0 131k 0 --:--:-- --:--:-- --:--:-- 131k

Note: In on-prem deployments, we can run the following command from the central infra VM.

curl -k -uadmin:< #enter elasticsearch password here#> https://<elastic search cluster ip:9200/_cat/indices?v | grep "red"

As seen in the above output, there are red indices in the database, which need to be deleted followed by a re-indexing of the red indices.

Note: Since a database is a critical component, performing a delete operation would require appropriate guidance. Contact Support for guidance.

 

Verification

After the red indices are deleted, check cluster health by using the following curl command. The cluster “status” should now be green.

curl -k -uadmin:<#enter elasticsearch password here#> https://<elastic search cluster ip >:9200/_cluster/health?pretty

{
  "cluster_name" : "YX38GA",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 2563,
  "active_shards" : 5127,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0