Support Support Downloads Knowledge Base Juniper Support Portal Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Contrail] Zookeeper fails to come up due to "Unable to load database on disk"

0

0

Article ID: KB36224 KB Last Updated: 24 Sep 2020Version: 1.0
Summary:

This article describes a workaround to address the issue of Zookeeper failing to come up and logging the "Unable to load database on disk" error.

Symptoms:

Users may observe that Zookeeper fails to come up and logs the "Unable to load database on disk" error. In zookeeper.log, an error sequence similar to the following may be seen:

2019-09-10 02:00:47,452 - ERROR [main:FileTxnSnapLog@210] - Parent /isr_change_notification missing for /isr_change_notification/isr_change_0000002662
2019-09-10 02:00:47,453 - ERROR [main:QuorumPeer@453]- Unable to load database on disk
java.io.IOException: Failed to process transaction type: 1 error: KeeperErrorCode = NoNode for /isr_change_notification
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:153)
        at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
        at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:417)
        at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:409)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:151)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /isr_change_notification
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:211)
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
        ... 6 more
2019-09-10 02:00:47,455 - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally
java.lang.RuntimeException: Unable to run quorum server
        at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:454)
       at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:409)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:151)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.io.IOException: Failed to process transaction type: 1 error: KeeperErrorCode = NoNode for /isr_change_notification
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:153)
        at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
        at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:417)
        ... 4 more
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /isr_change_notification
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:211)
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
        ... 6 more

 

Cause:

This error is due to Zookeeper's internal snapshot being wiped out or snapshot corruption.  

 

Solution:

On the Contrail node where Zookeeper fails to come up or keeps bouncing and the aforementioned error is observed in zookeeper.log, we can take the following actions to bring up the Zookeeper service:

  1. Make sure that the Zookeeper service is indeed shut down by service zookeeper stop.

  2.  Take a backup of the contents in /var/lib/zookeeper/version-2 to a temp location, and then remove the contents in /var/lib/zookeeper/version-2 by using rm -rf /var/lib/zookeeper/version-2/*.

  3. Restart the Zookeeper service by using service zookeeper start.

  4. Run service zookeeper status and monitor zookeeper.log to ensure that the service starts up correctly.

 

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search