Knowledge Search


×
 

[Junos Space] Database out of sync with error 2026 from "show slave status"

  [KB34899] Show Article Properties


Summary:

If a Junos Space deployment has been running on the same system with only upgrades and no rebuild for five years, the MySQL certificate has likely expired, preventing replication from working and reporting error 2026 (database out of sync) in the show slave status output.

Note: This article applies only for the error 2026 that is found in the show slave status output.

The article explains how to check whether the MySQL certificate has expired, regenerate the certificates, and restart the services.

Symptoms:

After seeing "Out Of Sync"  for the database node, perform the following:

Use the following command on the database nodes:

mysql -ujboss -p$(grep mysql.jboss /etc/sysconfig/JunosSpace/pwd | awk -F= '{print $2}') -e "show slave status\G show master status\G"

Note: This article applies only for the following error:

Last_IO_Errno: 2026
           Last_IO_Error: error connecting to master 'repUser@REMOVED:3306' - retry-time: 10  retries: 5

To confirm whether the certificate is the problem, use the following command on all database nodes:

openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/ca-cert.pem | grep After
openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/server-cert.pem | grep After

Output example: (This date should have passed.)

Not After : May 19 20:03:55 2019 GMT

Cause:

The MySQL certificates are generated for a period of five years. If your Junos Space system is upgraded across multiple versions without being rebuilt, the certificate will expire and go out of sync in the UI.

Solution:

Note: This process should be followed with JTAC assistance.

  1. Stop these processes on all Junos Space nodes by using the following commands:
service jmp-watchdog stop
service jboss stop
service jboss-dc stop
service mysql stop

Note: jboss-dc is active only on the VIP node, but the command will not do any harm.

  1. Confirm whether JBoss has been turned off.
service jboss status
service mysql status

The output should show a "not running" or stopped result.

  1. Make a note of the current expiration and file information to ensure that they have updated (all Junos Space database nodes).
openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/ca-cert.pem | grep After
openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/server-cert.pem | grep After
ls -l /var/chroot/mysql/etc/pki/mysql/
  1. Create a new root certificate [(only the first node (VIP)].

/var/www/cgi-bin/generateCACert.sh
  1. Confirm that new ca-cert.pem and ca-key.pem files have been generated. Compare to the output from the earlier step [only the first node (VIP)].

ls -l /var/chroot/mysql/etc/pki/mysql/
  1. Transfer the keypair to the other database node:
    1. From the VIP node, transfer the file as shown.

      scp /var/chroot/mysql/etc/pki/mysql/ca-cert.pem   admin@<other mysql node>:/home/admin/
      scp /var/chroot/mysql/etc/pki/mysql/ca-key.pem   admin@<other mysql node>:/home/admin/
    2. On the destination database node, move the file into place and fix permissions.

      mv /home/admin/ca-cert.pem /home/admin/ca-key.pem  /var/chroot/mysql/etc/pki/mysql/
      chown root:mysql /var/chroot/mysql/etc/pki/mysql/ca-cert.pem /var/chroot/mysql/etc/pki/mysql/ca-key.pem
      chmod 640 /var/chroot/mysql/etc/pki/mysql/ca-cert.pem /var/chroot/mysql/etc/pki/mysql/ca-key.pem
    3. Confirm that the new files ca-cert.pem and ca-key.pem exist on both nodes, and have the same size/timestamp.

      ls -l /var/chroot/mysql/etc/pki/mysql/  
  1. Create new server keys based on the root certificate on both database nodes:
cd /var/www/cgi-bin/secure
perl setupDbSSL operation=generateServerCert
ls -l /var/chroot/mysql/etc/pki/mysql/
  1. Check the expiration time on the certificates on all Junos Space database nodes:

openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/ca-cert.pem | grep After
openssl x509 -text -inform PEM -in /var/chroot/mysql/etc/pki/mysql/server-cert.pem | grep After

The dates should now be updated on all nodes. ca-cert will match all database nodes. server-cert will be different between both nodes.

  1. Now start services on the VIP node.
service jmp-watchdog start

Note: Watchdog starts the other processes.

Note: Additional database replication errors may be encountered after clearing this specific error. Database backup/restore, or other manual database resync processes will likely be needed to ensure proper replication. Database errors may not be seen for a few days until a conflicting data change has been made.”

  1. After you can log in to the WebUI from the VIP node, start all other Junos Space nodes.
service jmp-watchdog start
  1. Check MySQL status on both nodes to confirm that the error is no longer seen (can take some time after startup of the second node for the message to clear).

mysql -ujboss -p$(grep mysql.jboss /etc/sysconfig/JunosSpace/pwd | awk -F= '{print $2}') -e "show slave status\G show master status\G"

 
Related Links: