Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

CSO 5.1.1 - HA : VRR not accessible post server reboot

0

0

Article ID: KB35554 KB Last Updated: 06 Mar 2020Version: 1.0
Summary:

This KB article indicates VRR (Virtual Route Reflector) issue arises due to reboot of server in case of CSO HA 5.1.1 environment and a way to recover VRR.

Symptoms:
Tenant addition failure :
Task: 
Mar 5, 2020, 1:49:57 AMadding customer admin to tenant test4
Mar 5, 2020, 1:49:58 AMCreate tenant admin success.
Mar 5, 2020, 1:49:59 AMIAM granted group access to project test4 ccb4fcfb-32ab-41ff-9efd-b4008fbb9625
Mar 5, 2020, 1:50:02 AMCheck for customer object existence default-domain:test4:test4
Mar 5, 2020, 1:50:02 AMCreating customer default-domain:test4:test4
Mar 5, 2020, 1:50:02 AMDisable Root CA to customer ['default-domain', u'test4', u'test4']
Mar 5, 2020, 1:50:03 AMCreated customer default-domain:test4:test4 9a7fefdb-ab68-4dbc-994e-b6b6ef79dd9e
Mar 5, 2020, 1:50:04 AMOnboarding tenant in vims [] with tenant_type small
Mar 5, 2020, 1:50:07 AMadding properties to customer [u'default-domain', u'test4', u'test4']
Mar 5, 2020, 1:50:10 AMstarting Tenant creation in vrr
Mar 5, 2020, 1:55:39 AMTenant creation in VRR returned a failure status. Error message {u'status': u'FAILURE', u'output': u'. Traceback (most recent call last):\n File "/usr/local/lib/python2.7/site-packages/routing_manager_services/tasks/vrr_tenant_task.py", line 280, in vrr_tenant_op_aux\n create_vrr_tenant_task.tenant_vrr_operation(op_flag=create)\n File "/usr/local/lib/python2.7/site-packages/routing_manager_services/tasks/vrr_tenant_task.py", line 150, in tenant_vrr_operation\n raise Exception\nException\n', u'X-Request-Id': u'UI_e1f44dea-3729-8143-3471-9363a61f783d,4YOI,t0mZ', u'rpc_type': u'CREATE_VRR_TENANT', u'request_id': u'3fff5d16-b142-40ef-856d-585b02fb0bb1'}
Mar 5, 2020, 2:00:59 AMTask Fail Tenant creation in VRR returned a failure status. Error message {u'status': u'FAILURE', u'output': u'. Traceback (most recent call last):\n File "/usr/local/lib/python2.7/site-packages/routing_manager_services/tasks/vrr_tenant_task.py", line 280, in vrr_tenant_op_aux\n create_vrr_tenant_task.tenant_vrr_operation(op_flag=create)\n File "/usr/local/lib/python2.7/site-packages/routing_manager_services/tasks/vrr_tenant_task.py", line 150, in tenant_vrr_operation\n raise Exception\nException\n', u'X-Request-Id': u'UI_e1f44dea-3729-8143-3471-9363a61f783d,4YOI,t0mZ', u'rpc_type': u'CREATE_VRR_TENANT', u'request_id': u'3fff5d16-b142-40ef-856d-585b02fb0bb1'}
Mar 5, 2020, 2:00:59 AMTask complete
Mar 5, 2020, 2:00:59 AMTask Fail Tenant creation in VRR returned a failure status. Error message {u'status': u'FAILURE', u'output': u'. Traceback (most recent call last):\n File "/usr/local/lib/python2.7/site-packages/routing_manager_services/tasks/vrr_tenant_task.py", line 280, in vrr_tenant_op_aux\n create_vrr_tenant_task.tenant_vrr_operation(op_flag=create)\n File "/usr/local/lib/python2.7/site-packages/routing_manager_services/tasks/vrr_tenant_task.py", line 150, in tenant_vrr_operation\n raise Exception\nException\n', u'X-Request-Id': u'UI_e1f44dea-3729-8143-3471-9363a61f783d,4YOI,t0mZ', u'rpc_type': u'CREATE_VRR_TENANT', u'request_id': u'3fff5d16-b142-40ef-856d-585b02fb0bb1'}
Mar 5, 2020, 2:00:59 AMJob Fail Unknown id: Error: oper 3 url /tssm/customer/9a7fefdb-ab68-4dbc-994e-b6b6ef79dd9e body {"customer":{"state": "failed", "uuid": "9a7fefdb-ab68-4dbc-994e-b6b6ef79dd9e"}} response {"status_code": "404", "error_tag": "No ID Found", "error_message": "The resource with id 9a7fefdb-ab68-4dbc-994e-b6b6ef79dd9e of type customer does not exist", "error_app_message": "Unknown id: 9a7fefdb-ab68-4dbc-994e-b6b6ef79dd9e", "error_diag": "This error occurs when the resource with the specified id is not found", "error_code": "40027"}
Solution:
  1. Check the status of all vm using command - virsh list --all and vrr should be in running status
  2. Run  ./components_health.sh - to check health status of rest of components.
  3. In certain scenarios listed in KB35551 rabbitmq can be in unhealthy state post reboot of server
  4. Login to startup server and navigate to CSO install directory
  5. Run recovery of rabbitmq first ( if rabbitmq is in unhealthy state)
  6. Recovery of rabbitmq should restart central and regional msvm ( Refer KB35551)
  7. Run recovery for vrr
root@startupserver1:/Contrail_Service_Orchestration_5.1.1# ./recovery.sh
***************
This tool assists you recover your CSO setup.
***************


Following components can be recovered

1: mariadb
2: vrr
3: icinga
4: rabbitmq

Specify one of the component to recover (In Number) : 2
INFO     Started recovering vrr component at 2020-03-05 11:54:42.336216 ...
INFO     VRR recovery is initiated...
ERROR    Vrr - X.X.X.X is unhealthy
ERROR    Vrr - X.X.X.X is unhealthy
INFO     Recovery takes time, please be patient
INFO     VRR recovery started. Please wait...
INFO     VRR config sync job has been initiated, please monitor from CSO UI for the status of the job...
INFO     Completed recovering vrr component at 2020-03-05 11:58:01.668640 .
INFO     Time taken to recover 0:03:19.332424
Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search