Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Contrail] Zookeeper end of stream exception

0

0

Article ID: KB35220 KB Last Updated: 16 Nov 2019Version: 1.0
Summary:

When searching the zookeeper log file, "end of stream exception" may be observed. This exception is an indicator of the end of the data stream of a session. It usually occurs when closing a connection to zookeeper. This article provides an example on how it is triggered.  

Symptoms:

During zookeeper troubleshooting, the following exception may be seen in zookeeper.log. This exception does not signal a defect on the zookeeper side. Instead, it shows a connection to client is reset or closed. 

EndOfStreamException: Unable to read additional data from client sessionid 0x16e18e9152f002f, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:745)
Solution:

Lets use svc-monitor as an example. For a svc-monitor, a session is established with zookeeper, and a znode will be created for svc-monitor inside the zookeeper. The data part of this znode will be pid of svc-monitor.

Checking pid and status on contrail node.

# contrail-status -d | grep svc
contrail-svc-monitor       active       pid 8785, uptime 0:01:36

Use the command, '/usr/share/zookeeper/bin/zkCli.sh' to enter zookeeper CLI. Then look for children znode created under /svc-monitor, one of which should have data matching pid of this svc-monitor. The ephemeralOwner of this znode will be session id of this svc-monitor.

# /usr/share/zookeeper/bin/zkCli.sh
Connecting to localhost:2181
<snipped>
[zk: localhost:2181(CONNECTED) 1] get /svc-monitor/02efe5df71eb4613a3591c86a0c684d6__lock__0000000007
8785
cZxid = 0x595
ctime = Tue Oct 29 13:44:00 PDT 2019
mZxid = 0x595
mtime = Tue Oct 29 13:44:00 PDT 2019
pZxid = 0x595
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x16e18e9152f002f
dataLength = 4
numChildren = 0

If svc-monitor is restarted, we can see corresponding log event showing in zookeeper log. 

# service contrail-svc-monitor restart 
contrail-svc-monitor: stopped
contrail-svc-monitor: started

Notice this log event also provides session ID information, so we can confirm this exception is caused by svc-monitor restarting and disconnecting operation. 

2019-10-29 13:45:58,543 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x16e18e9152f002f, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:745)
2019-10-29 13:45:58,544 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /10.176.0.250:48980 which had sessionid 0x16e18e9152f002f

For more on troubleshooting Zookeeper, refer to KB31144 - Contrail Getting Started - Administration, Configuration & Troubleshooting (JumpStation)

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search