[Junos-SRX] SRX device goes "Out-Of-Sync" in Junos Space after configuration push

  [KB32709] Show Article Properties


Summary:

This article explains why an SRX device is reported to be "Out-Of-Sync" in Junos Space after a configuration push from Junos Space to the SRX cluster.

 

Symptoms:

The issue is seen when all of the following conditions are met:

  • When the configuration push is initiated from Junos Space to the SRX device

  • When the SRX device has a syslog configuration with "EVENT" mode logging and also has traffic logs enabled

When an attempt is made to push configuration changes from Junos Space, the configuration push completes on the SRX device Command Line Interface (CLI), but the SRX cluster goes "Out-Of-Sync" and throws a commit error on Junos Space.

 

Cause:

The main reason for this is as follows:

When a commit from Junos Space completes, the SRX device logs a "commit complete" message in syslog.

UI_COMMIT_PROGRESS [junos@xxxxx message="commit complete"] Commit operation in progress: commit complete 

But given the current eventd architecture, it is not guaranteed that all logs will reach eventd, particularly when there is excessive logging of messages. And Junos Space relies entirely on eventd messages to synchronize a device. Consequently, if a "commit complete" message is omitted from the SRX device's syslog, Junos Space marks the device as "Out-Of-Sync."

 

Solution:

There are two ways to resolve this problem:

  1. The SRX cluster must be re-synched in Junos Space. This can be done by modifying the logging mode from "EVENT" to "STREAM," which reduces the pressure on the Routing Engine (RE). When the mode is thus modified, the probability of the RE omitting syslog messages is reduced, leading to a proper sync of the SRX device with Junos Space.

  2. Another workaround that can be implemented is to increase the eventd buffer as follows:

# set event-options max-events-queued 1000

(This is a hidden command, so do ensure that it is typed completely.)

The above command will read the events from sockbuf and delay processing of syslog events. It could be helpful in this case because sockbuf will be cleared quickly by eventd. But this does not guarantee that the issue may not occur again. Under excessive logging, syslog messages may still get dropped.

 

Related Links: