Support Support Downloads Knowledge Base Apex Support Portal Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[SBR] Commands to troubleshoot NDB issues in the SBR Carrier Cluster

0

0

Article ID: KB27008 KB Last Updated: 08 Mar 2017Version: 3.0
Summary:
This article provides information about the commands that can be used to troubleshoot NDB issues in the SBR Carrier cluster.
Symptoms:
At times, when there are issues with the SSR cluster, the individual responsible for SBR management is unable to isolate the issue to the SBR API node or one of the NDB nodes. Along with the ndb_error_reporter output, this article provides certain commands that can determine the source of SSR issues.
Cause:
 
Solution:
ndb_mgm commands:  

These commands are run by signing in as hadm and issuing the ndb_mgm command to get to the NDB management interface:
-bash-3.00# su - hadm
hadm@mzone-1:~> ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm>
ALL REPORT MEMORY: This command displays how much data memory and index memory is being used by each data node:
ndb_mgm> ALL REPORT MEMORY
Connected to Management Server at: 10.17.14.223:5235
Node 1: Data usage is 0%(102 32K pages of total 115200)
Node 1: Index usage is 0%(127 8K pages of total 51264)
Node 2: Data usage is 2%(3096 32K pages of total 115200)
Node 2: Index usage is 0%(127 8K pages of total 51264)
ALL DUMP 1000: This command generates the same information as the REPORT MEMORY command  to the Cluster logs, which are included in ndb_error_reporter.

ALL CLUSTERLOG STATISTICS=15: This command adds more statistics to the cluster logs (7 is the default value). This should be run prior to gathering ndb_error_reporter, as the reporter will gather the cluster logs.

Note: Setting the threshold for STATISTICS to 15 causes the cluster log to become very verbose, and grow quite rapidly in size, in direct proportion to the number of cluster nodes and the amount of activity in the MySQL Cluster.

MySQL commands:

These commands are run via the MySQL API by logging in as hadm and issuing the mysql command:

-bash-3.00# su - hadm
hadm@mzone-1:~> mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 55
Server version: 5.1.56-ndb-7.1.15a-br14230128-cluster-com MySQL Cluster Server (Commercial)

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
This is commercial software, and use of this software is governed
by your applicable license agreement with MySQL

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

SHOW FULL PROCESSLIST: This command displays the threads that are running and their statuses:

mysql> SHOW FULL PROCESSLIST;
+----+-------------+-----------+------+---------+------+-----------------------------------+-----------------------+
| Id | User        | Host      | db   | Command | Time | State                             | Info                  |
+----+-------------+-----------+------+---------+------+-----------------------------------+-----------------------+
| 1  | system user |           |      | Daemon  | 0    | Waiting for event from ndbcluster | NULL                  |
| 55 | hadmsql     | localhost | NULL | Query   | 0    | NULL                              | SHOW FULL PROCESSLIST |
+----+-------------+-----------+------+---------+------+-----------------------------------+-----------------------+
2 rows in set (0.00 sec)

SHOW [GLOBAL] STATUS: This command provides the server status information. Can be used with LIKE 'NDB% for the specific NDB status:

mysql> SHOW GLOBAL STATUS LIKE 'NDB%';
+--------------------------------------------+--------------+
| Variable_name                              | Value        |
+--------------------------------------------+--------------+
| Ndb_api_event_data_count_injector          | 53           |
| Ndb_api_event_nondata_count_injector       | 4            |
| Ndb_api_event_bytes_count_injector         | 47984        |
| Ndb_api_wait_exec_complete_count_slave     | 0            |
| Ndb_api_wait_scan_result_count_slave       | 0            |
| Ndb_api_wait_meta_request_count_slave      | 0            |
| Ndb_api_wait_nanos_count_slave             | 0            |
| Ndb_api_bytes_sent_count_slave             | 0            |
| Ndb_api_bytes_received_count_slave         | 0            |
| Ndb_api_trans_start_count_slave            | 0            |
| Ndb_api_trans_commit_count_slave           | 0            |
| Ndb_api_trans_abort_count_slave            | 0            |
| Ndb_api_trans_close_count_slave            | 0            |
| Ndb_api_pk_op_count_slave                  | 0            |
| Ndb_api_uk_op_count_slave                  | 0            |
| Ndb_api_table_scan_count_slave             | 0            |
| Ndb_api_range_scan_count_slave             | 0            |
| Ndb_api_pruned_scan_count_slave            | 0            |
| Ndb_api_scan_batch_count_slave             | 0            |
| Ndb_api_read_row_count_slave               | 0            |
| Ndb_api_trans_local_read_row_count_slave   | 0            |
| Ndb_api_wait_exec_complete_count           | 130          |
| Ndb_api_wait_scan_result_count             | 221          |
| Ndb_api_wait_meta_request_count            | 481          |
| Ndb_api_wait_nanos_count                   | 7874490100   |
| Ndb_api_bytes_sent_count                   | 201828       |
| Ndb_api_bytes_received_count               | 306592       |
| Ndb_api_trans_start_count                  | 97           |
| Ndb_api_trans_commit_count                 | 52           |
| Ndb_api_trans_abort_count                  | 2            |
| Ndb_api_trans_close_count                  | 97           |
| Ndb_api_pk_op_count                        | 2119         |
| Ndb_api_uk_op_count                        | 2            |
| Ndb_api_table_scan_count                   | 40           |
| Ndb_api_range_scan_count                   | 3            |
| Ndb_api_pruned_scan_count                  | 0            |
| Ndb_api_scan_batch_count                   | 62           |
| Ndb_api_read_row_count                     | 2592         |
| Ndb_api_trans_local_read_row_count         | 1219         |
| Ndb_api_event_data_count                   | 53           |
| Ndb_api_event_nondata_count                | 4            |
| Ndb_api_event_bytes_count                  | 47984        |
| Ndb_cluster_node_id                        | 61           |
| Ndb_config_from_host                       | 10.17.14.223 |
| Ndb_config_from_port                       | 5235         |
| Ndb_number_of_data_nodes                   | 2            |
| Ndb_number_of_ready_data_nodes             | 2            |
| Ndb_connect_count                          | 0            |
| Ndb_execute_count                          | 0            |
| Ndb_scan_count                             | 0            |
| Ndb_pruned_scan_count                      | 0            |
| Ndb_api_wait_exec_complete_count_session   | 0            |
| Ndb_api_wait_scan_result_count_session     | 0            |
| Ndb_api_wait_meta_request_count_session    | 0            |
| Ndb_api_wait_nanos_count_session           | 0            |
| Ndb_api_bytes_sent_count_session           | 0            |
| Ndb_api_bytes_received_count_session       | 0            |
| Ndb_api_trans_start_count_session          | 0            |
| Ndb_api_trans_commit_count_session         | 0            |
| Ndb_api_trans_abort_count_session          | 0            |
| Ndb_api_trans_close_count_session          | 0            |
| Ndb_api_pk_op_count_session                | 0            |
| Ndb_api_uk_op_count_session                | 0            |
| Ndb_api_table_scan_count_session           | 0            |
| Ndb_api_range_scan_count_session           | 0            |
| Ndb_api_pruned_scan_count_session          | 0            |
| Ndb_api_scan_batch_count_session           | 0            |
| Ndb_api_read_row_count_session             | 0            |
| Ndb_api_trans_local_read_row_count_session | 0            |
| Ndb_conflict_fn_max                        | 0            |
| Ndb_conflict_fn_old                        | 0            |
| Ndb_conflict_fn_max_del_win                | 0            |
| Ndb_index_stat_cache_query                 | 0            |
| Ndb_index_stat_cache_clean                 | 0            |
+--------------------------------------------+--------------+
74 rows in set (0.01 sec) 

SHOW ENGINE NDBCLUSTER STATUS: The output of this statement contains information about the server's connection to the cluster, creation, usage of MySQL Cluster objects, and binary logging for the MySQL Cluster replication:

mysql> SHOW ENGINE NDBCLUSTER STATUS;
+------------+-----------------------+--------------------------------------------------------------------+
| Type        | Name                    | Status |
+------------+-----------------------+--------------------------------------------------------------------+
| ndbcluster | connection        | cluster_node_id=61, connected_host=10.17.14.223, 
connected_port=5235, number_of_data_nodes=2, number_of_ready_data_nodes=2, connect_count=0 |
| ndbcluster | NdbTransaction | created=2, free=2, sizeof=368 |
| ndbcluster | NdbOperation | created=4, free=4, sizeof=952 |
| ndbcluster | NdbIndexScanOperation | created=0, free=0, sizeof=1160 |
| ndbcluster | NdbIndexOperation | created=0, free=0, sizeof=960 |
| ndbcluster | NdbRecAttr | created=0, free=0, sizeof=88 |
| ndbcluster | NdbApiSignal | created=16, free=16, sizeof=144 |
| ndbcluster | NdbLabel | created=0, free=0, sizeof=200 |
| ndbcluster | NdbBranch | created=0, free=0, sizeof=32 |
| ndbcluster | NdbSubroutine | created=0, free=0, sizeof=72 |
| ndbcluster | NdbCall | created=0, free=0, sizeof=24 |
| ndbcluster | NdbBlob | created=0, free=0, sizeof=496 |
| ndbcluster | NdbReceiver | created=0, free=0, sizeof=136 |
| ndbcluster | NdbLockHandle | created=0, free=0, sizeof=48 |
| ndbcluster | binlog | latest_epoch=10864119775232005, latest_trans_epoch=10857793288404994, 
latest_received_binlog_epoch=0, latest_handled_binlog_epoch=0, latest_applied_binlog_epoch=0 |
+------------+-----------------------+---------------------------------------------------------------------+
15 rows in set (0.00 sec) 

SHOW [GLOBAL] VARIABLES: This command can be used with LIKE 'NDB%' for NDB specific system variables:

mysql> SHOW GLOBAL VARIABLES LIKE 'NDB%';
+-------------------------------------+---------------------------+
| Variable_name | Value |
+-------------------------------------+---------------------------+
| ndb_autoincrement_prefetch_sz | 1 |
| ndb_batch_size | 32768 |
| ndb_blob_read_batch_bytes | 65536 |
| ndb_blob_write_batch_bytes | 65536 |
| ndb_cache_check_time | 0 |
| ndb_cluster_connection_pool | 1 |
| ndb_connectstring | 10.17.14.223:5235,10.17.14.224:5235;nodeid=61 |
| ndb_deferred_constraints | 0 |
| ndb_distribution | KEYHASH |
| ndb_extra_logging | 1 |
| ndb_force_send | ON |
| ndb_index_stat_cache_entries | 32 |
| ndb_index_stat_enable | OFF |
| ndb_index_stat_option | loop_checkon=1000ms,loop_idle=1000ms,loop_busy=100ms,update_batch=1,
read_batch=4,idle_batch=32,check_batch=32,check_delay=1m,delete_batch=8,clean_delay=0,error_batch=4,
error_delay=1m,evict_batch=8,evict_delay=1m,cache_limit=32M,cache_lowpct=90 |
| ndb_index_stat_update_freq | 20 |
| ndb_log_apply_status | OFF |
| ndb_log_bin | ON |
| ndb_log_binlog_index | ON |
| ndb_log_empty_epochs | OFF |
| ndb_log_orig | OFF |
| ndb_log_update_as_write | ON |
| ndb_log_updated_only | ON |
| ndb_mgmd_host | 10.17.14.223:5235,10.17.14.224:5235;nodeid=61 |
| ndb_nodeid | 0 |
| ndb_optimization_delay | 10 |
| ndb_optimized_node_selection | 3 |
| ndb_report_thresh_binlog_epoch_slip | 3 |
| ndb_report_thresh_binlog_mem_usage | 10 |
| ndb_table_no_logging | OFF |
| ndb_table_temporary | OFF |
| ndb_use_copying_alter_table | OFF |
| ndb_use_exact_count | OFF |
| ndb_use_transactions | ON |
| ndb_wait_connected | 0 |
| ndb_wait_setup | 15 |
| ndbinfo_database | ndbinfo |
| ndbinfo_max_bytes | 0 |
| ndbinfo_max_rows | 10 |
| ndbinfo_offline | OFF |
| ndbinfo_show_hidden | OFF |
| ndbinfo_table_prefix | ndb$ |
| ndbinfo_version | 459023 |
+-------------------------------------+
42 rows in set (0.00 sec)
Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search