Support Support Downloads Knowledge Base Juniper Support Portal Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[PTX] ST gets stucked due to increment of FD count reaching its limit

0

0

Article ID: KB34657 KB Last Updated: 08 Jul 2019Version: 1.0
Summary:

Telemetry stops working and no data is sent. Each time when subscription occurs, agentd opens a FD to read. The total count of FDs of agentd keeps increasing until SDN-Telemetry process gets restarted. This Issue can be observed on 17.2X75-D103.3 and has been fixed in later releases.

Symptoms:

The command 'show agent sensors' does not indicate a cause. The log messages below shows 'Too many open files'.

Verify => procstat -f <agentd.pid>                 
root@re0:/var/home/remote # procstat -f 6819
  PID COMM               FD T V FLAGS     REF  OFFSET PRO NAME        
 6819 agentd            text v r r-------  -       - -   /packages/mnt/junos-runtime-mx/usr/sbin/agentd
 6819 agentd             cwd v d r-------  -       - -   /var/tmp          
 6819 agentd            root v d r-------  -       - -   /                 
 6819 agentd            jail v d r-------  -       - -   /                 
 6819 agentd               0 v c rw------  3       0 -   /dev/null         
 6819 agentd               1 v c rw------  3       0 -   /dev/null         
 6819 agentd               2 v c rw------  3       0 -   /dev/null         
 6819 agentd               3 s - rw------  1       0 UDD /var/run/logpriv
 6819 agentd               4 v r rw-----l 68       5 -   /var/run/jlaunchd.pid
 6819 agentd               5 v r rw-----l  1       5 -   /var/run/agentd.pid
 6819 agentd               6 s - rw---n--  1       0 UDS /var/run/agentd_mgmt
 6819 agentd               7 p - rw---n--  1       0 -   -                 
 6819 agentd               8 p - rw------  1       0 -   -                 
 6819 agentd               9 v r r-------  1      16 -   -                 
 6819 agentd              10 p - rw---n--  1       0 -   -                 
 6819 agentd              11 p - rw------  1       0 -   -                 
 6819 agentd              12 s - rw------  1       0 UDD /var/run/eventd_events
 6819 agentd              13 s - rw---n--  1       0 ?    
 6819 agentd              14 v c r-------  1     320 -   /dev/random       
 6819 agentd              15 s - rw---n--  1       0 ?    
 6819 agentd              16 s - rw---n--  1       0 ?    
 6819 agentd              17 s - rw---n--  1       0 TCP 0.0.0.0:9500 0.0.0.0:0
 6819 agentd              18 s - rw---n--  1       0 UDP 0.0.0.0:2000 0.0.0.0:0
 6819 agentd              19 p - rw---n--  1       0 -   -                 
 6819 agentd              20 p - rw---n--  1       0 -   -                 
 6819 agentd              21 s - rw---n--  1       0 UDS -
 6819 agentd              22 s - rw---n--  1       0 UDS -
 6819 agentd              23 v r r-------  1      16 -   -                 
 6819 agentd              24 s - rw---n--  1       0 TCP 128.0.0.4:54567 128.0.0.4:51883
 6819 agentd              25 s - rw---n--  1       0 TCP 128.0.0.4:9500 128.0.0.4:64870
 6819 agentd              26 s - rw---n--  1       0 UDS /var/run/agentd-26
 6819 agentd              29 v r r-------  1      16 -   -                 
 6819 agentd              30 v r r-------  1      16 -   -                 
 6819 agentd              31 v r r-------  1      16 -   -                 
 6819 agentd              32 v r r-------  1      16 -   -                 
 6819 agentd              33 v r r-------  1      16 -   -                 
 6819 agentd              34 v r r-------  1      16 -   -                 
 6819 agentd              35 v r r-------  1      16 -   -                 
 6819 agentd              36 v r r-------  1      16 -   -                 
 6819 agentd              37 v r r-------  1      16 -   -                 
 6819 agentd              38 v r r-------  1      16 -   -                 
 6819 agentd              39 v r r-------  1      16 -   -                 
 6819 agentd              40 v r r-------  1      16 -   -                 
 6819 agentd              41 v r r-------  1      16 -   -                 
 6819 agentd              42 v r r-------  1      16 -   -                 
 6819 agentd              43 v r r-------  1      16 -   -                 
 6819 agentd              44 v r r-------  1      16 -   -                 
 6819 agentd              45 v r r-------  1      16 -   -                 
 6819 agentd              46 v r r-------  1      16 -   -                 
 6819 agentd              47 v r r-------  1      16 -   -                 
 6819 agentd              48 v r r-------  1      16 -   -                 
 6819 agentd              49 v r r-------  1      16 -   -                 
 6819 agentd              50 v r r-------  1      16 -   -                 
 6819 agentd              51 v r r-------  1      16 -   -                 
 6819 agentd              52 v r r-------  1      16 -   -                 
 6819 agentd              53 v r r-------  1      16 -   -                 
 6819 agentd              54 v r r-------  1      16 -   -                 
 6819 agentd              55 v r r-------  1      16 -   -                 
 6819 agentd              56 v r r-------  1      16 -   -                 
 6819 agentd              57 v r r-------  1      16 -   -                 
 6819 agentd              58 v r r-------  1      16 -   -                 
 6819 agentd              59 v r r-------  1      16 -   -                 
 6819 agentd              60 v r r-------  1      16 -   -                 
 6819 agentd              61 v r r-------  1      16 -   -                 
 6819 agentd              62 v r r-------  1      16 -   -                 
 6819 agentd              63 v r r-------  1      16 -   -                 
 6819 agentd              64 v r r-------  1      16 -   -                 
 6819 agentd              65 v r r-------  1      16 -   -                 
 6819 agentd              66 v r r-------  1      16 -   -                 
 6819 agentd              67 v r r-------  1      16 -   -                 
 6819 agentd              68 v r r-------  1      16 -   -                 
 6819 agentd              69 v r r-------  1      16 -   -                 
 6819 agentd              70 v r r-------  1      16 -   -                 
 6819 agentd              71 v r r-------  1      16 -   -                 
 6819 agentd              72 v r r-------  1      16 -   -                 
 6819 agentd              73 v r r-------  1      16 -   -                 
 6819 agentd              74 v r r-------  1      16 -   -                 
 6819 agentd              75 v r r-------  1      16 -   -                 
 6819 agentd              76 v r r-------  1      16 -   -                 
 6819 agentd              77 v r r-------  1      16 -   -                 
 6819 agentd              78 v r r-------  1      16 -   -                 
 6819 agentd              79 v r r-------  1      16 -   -                 
 6819 agentd              80 v r r-------  1      16 -   -                 
 6819 agentd              81 v r r-------  1      16 -   -                 
 6819 agentd              82 v r r-------  1      16 -   -                 
 6819 agentd              83 v r r-------  1      16 -   -                 
 6819 agentd              84 v r r-------  1      16 -   -                 
 6819 agentd              85 v r r-------  1      16 -   -                 
 6819 agentd              86 v r r-------  1      16 -   -                 
 6819 agentd              87 v r r-------  1      16 -   -                 
 6819 agentd              88 v r r-------  1      16 -   -                 
 6819 agentd              89 v r r-------  1      16 -   -                 
 6819 agentd              90 v r r-------  1      16 -   -                 
 6819 agentd              91 v r r-------  1      16 -   -                 
 6819 agentd              92 v r r-------  1      16 -   -        

Log:

Feb 24 05:37:43  re0 mosquitto[6710]: Socket error on client client-1, disconnecting.
Feb 24 05:37:35  re0 mgd[84106]: UI_COMMIT_PROGRESS: Commit operation in progress:  notifying none(0)
Feb 24 05:37:35  re0 mgd[84106]: UI_COMMIT_PROGRESS: Commit operation in progress: cannot signal 'None process', signal 1, no pid
Feb 24 05:37:35  re0 mgd[84106]: UI_COMMIT_PROGRESS: Commit operation in progress:  notifying agentd(18)
Feb 24 05:37:35  re0 mgd[84106]: UI_COMMIT_PROGRESS: Commit operation in progress: signaling 'agent for all the telemetry sensors', pid 6720, signal 1, status 0 with notification errors enabled
Feb 24 05:37:35  re0 agentd[6720]: agentd_sighup_proc: AgentD received SIGHUP. is_master:1
Feb 24 05:37:35  re0 mgd[84106]: UI_EPHEMERAL_COMMIT_COMPLETED: commit complete on 'junos-analytics' ephemeral database
Feb 24 05:37:35  re0 agentd[6720]: UI_DBASE_OPEN_FAILED: Database open failed for file '/var/run/db/juniper.eph_junos-analytics': Too many open files
Feb 24 05:37:35  re0 agentd[6720]: agentd_config_read_traceoptions 94: /var/log/agentd  /var/log/agentd 0 0 0
Feb 24 05:37:35  re0 agentd[6720]: agentd_config_read: Entering config read. is_master:1
Feb 24 05:37:35  re0 mgd-api[6510]: MGD_API_APPLICATION_DISCONNECT: Client  disconnected from mgd-api
Feb 24 05:37:43  re0 sshd[84115]: Did not receive identification string from 10.170.248.68
Feb 24 05:37:43  re0 inetd[6456]: /usr/sbin/sshd[84115]: exited, status 255
Feb 24 05:37:43  re0 mosquitto[6710]: Socket error on client client-1, disconnecting.
Cause:

Each time when subscription occurs, agentd opens the FD to read and max open FD total count is 1024. There was an FD leak and when FD reaches to 1024, agentd stops reading the configuration.

Solution:

Restarting SDN-telemetry will resolve the problem. However, this issue has been fixed in releases after 17.2X75-D103.3.

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search