Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Junos Space] Database out of sync with error 2026 from "show slave status"

2

0

Article ID: KB34899 KB Last Updated: 13 Feb 2020Version: 2.0
Summary:

If a Junos Space deployment has been running on the same system with only upgrades and no rebuild for five years, the MySQL certificate has likely expired, preventing replication from working and reporting error 2026 (database out of sync) in the show slave status output.

Note: This article applies only for the error 2026 that is found in the show slave status output.

The article explains how to check whether the MySQL certificate has expired, regenerate the certificates, and restart the services.

Symptoms:

After seeing "Out Of Sync"  for the database node, perform the following:

Use the following command on the database nodes:

mysql -ujboss -p$(grep mysql.jboss /etc/sysconfig/JunosSpace/pwd | awk -F= '{print $2}') -e "show slave status\G show master status\G"

Note: This article applies only for the following error:

Last_IO_Errno: 2026
           Last_IO_Error: error connecting to master 'repUser@REMOVED:3306' - retry-time: 10  retries: 5

To confirm whether the certificate is the problem, use the following command on all database nodes:

openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/ca-cert.pem | grep After
openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/server-cert.pem | grep After

Output example: (This date should have passed.)

Not After : May 19 20:03:55 2019 GMT

If this date is NOT in the past.  This KB will NOT correct the problem.

Cause:

The MySQL certificates are generated for a period of five years. If your Junos Space system is upgraded across multiple versions without being rebuilt, the certificate will expire and go out of sync in the UI.

Solution:

Note: This process should be followed with JTAC assistance.
Note:  Do not allow the VIP to move. Ensure VIP has not moved nodes since this error started appearing

  1. Stop these processes on all Junos Space nodes by using the following commands:

service jmp-watchdog stop
service jboss stop
service jboss-dc stop
service mysql stop

Note: jboss-dc is active only on the VIP node, but the command will not do any harm.

  1. Confirm whether JBoss has been turned off.

service jboss status
service mysql status

The output should show a "not running" or stopped result.

  1. Make a note of the current expiration and file information to ensure that they have updated (all Junos Space database nodes).

openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/ca-cert.pem | grep After
openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/server-cert.pem | grep After
ls -l /var/chroot/mysql/etc/pki/mysql/
  1. Create a new root certificate [(only the first node (VIP)].

/var/www/cgi-bin/generateCACert.sh
  1. Confirm that new ca-cert.pem and ca-key.pem files have been generated. Compare to the output from the earlier step [only the first node (VIP)].

ls -l /var/chroot/mysql/etc/pki/mysql/
  1. Transfer the keypair to the other database node:

    1. From the VIP node, transfer the file as shown.

      scp /var/chroot/mysql/etc/pki/mysql/ca-cert.pem   admin@<other mysql node>:/home/admin/
      scp /var/chroot/mysql/etc/pki/mysql/ca-key.pem   admin@<other mysql node>:/home/admin/
    2. On the destination database node, move the file into place and fix permissions.

      mv /home/admin/ca-cert.pem /home/admin/ca-key.pem  /var/chroot/mysql/etc/pki/mysql/
      chown root:mysql /var/chroot/mysql/etc/pki/mysql/ca-cert.pem /var/chroot/mysql/etc/pki/mysql/ca-key.pem
      chmod 640 /var/chroot/mysql/etc/pki/mysql/ca-cert.pem /var/chroot/mysql/etc/pki/mysql/ca-key.pem
    3. Confirm that the new files ca-cert.pem and ca-key.pem exist on both nodes, and have the same size/timestamp.

      ls -l /var/chroot/mysql/etc/pki/mysql/  
  1. Create new server keys based on the root certificate on both database nodes:

cd /var/www/cgi-bin/secure
perl setupDbSSL operation=generateServerCert
ls -l /var/chroot/mysql/etc/pki/mysql/
  1. Check the expiration time on the certificates on all Junos Space database nodes:

openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/ca-cert.pem | grep After
openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/server-cert.pem | grep After

The dates should now be updated on all nodes. ca-cert will match all database nodes. server-cert will be different between both nodes.

  1. Now start services on the VIP node.

service jmp-watchdog start

Note: Watchdog starts the other processes.

Note: Additional database replication errors may be encountered after clearing this specific error. Database backup/restore, or other manual database resync processes will likely be needed to ensure proper replication. Database errors may not be seen for a few days until a conflicting data change has been made.”

  1. After you can log in to the WebUI from the VIP node, start all other Junos Space nodes.

service jmp-watchdog start
  1. Check MySQL status on both nodes to confirm that the error is no longer seen (can take some time after startup of the second node for the message to clear).

    IF the error has been occurring  for more than 24 hrs, perform a database backup and restore. To completely sync the data.

    If  the error has only existed for 1 day or less, you can wait for the system to catch up on replication. (non-replicated bin-files are likely lost if more than 24hrs)

mysql -ujboss -p$(grep mysql.jboss /etc/sysconfig/JunosSpace/pwd | awk -F= '{print $2}') -e "show slave status\G show master status\G"

 
Modification History:

2020-02-13: Added some clarification/warnings notes.

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search