Knowledge Search


×
 

[Junos Space] Data is inconsistant between sites after DR Failover

  [KB33789] Show Article Properties


Summary:

When Junos Space Platform is configured for DR operations, the command 'jmp-dr start'  backs up the space database on the active site, and restores it on the second.

Under certain conditions, the backup is incomplete and the partial backup is restored on the standby DR site.  The commands do not report any error occurred to the user.

This issue is seen in Junos Space Platform 17.2R1, fixed in version 18.3R1, and likely impacts all versions before 18.3R1.

Symptoms:

When failing over to DR site, data is incorrect or missing.  [Do NOT start DR with the bad site as the active, failover is ok]

Check if the backup finished:

  1. Look at the log, /var/log/jmp-geo/dr-cli-start.log on the (Original) Active/VIP for lines similar to:

    Tue Aug  7 07:23:51 2018: start MySQL dump ------ begin
    <Other Line removed>
    mysqldump: Error 2020: Got packet bigger than 'max_allowed_packet' bytes when dumping table `IdpVersionsEntity` at row: 0
    Tue Aug  7 07:24:44 2018: start MySQL dump ------ end: 0

    If similar error is seen, you have encountered the issue, the DB backup is incomplete.

  2. To manually see if the DB backup finished, from the VIP node of the Active DR site:

    zcat /var/cache/jmp-geo/data/db.gz | tail -n 5

    Successful DB backup for DR start will end with a line like below:

    -- Dump completed on 2018-08-07 21:13:52
    

    Failed backup, will be ANY other line as the last

Cause:

DR Start backup script has no error checking for the DB backup command. Due to the size of some of the data, the backup may be incomplete.

This is the most likely cause and workaround.  After applying this workaround, if the database backup continues to be an incomplete file, contact your JTAC representative.

Solution:

This issue can be resolved by increasing the max_allowed_packet value for the mysqldump utility command used by the DR start command.

  1. Stop DR (One node):

    jmp-dr stop
  2. Increase the max_allowed_packet size for DR utility scripts (Both sites, All Space JBoss and DB nodes):

    1. Check whether the utility already has max_allowed_packet option for mysqldump command as below on all the nodes on both sites:

      grep -oP '(?<=max_allowed_packet=)[0-9]*G'   /var/cache/jmp-geo/backup/script/backupReal.sh /usr/nma/lib/DrUtil.pm
    2. If there is NO output, the config needs to be added with the 2 commands:

      sed -i.bak -e 's/mysqldump/mysqldump --max_allowed_packet=2G/g' /usr/nma/lib/DrUtil.pm
      sed -i.bak -e 's/mysqldump/mysqldump --max_allowed_packet=2G/g' /var/cache/jmp-geo/backup/script/backupReal.sh
    3. After these 2 commands, use the show command again:

      grep -oP '(?<=max_allowed_packet=)[0-9]*G'    /var/cache/jmp-geo/backup/script/backupReal.sh /usr/nma/lib/DrUtil.pm

    4. You should see output similar to:

      /var/cache/jmp-geo/backup/script/backupReal.sh:2G
      /usr/nma/lib/DrUtil.pm:2G
  3. Start DR

    jmp-dr start
Related Links: