Support Support Downloads Knowledge Base Apex Support Portal Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[PTX] Commit fails with 'Please check space available in /var/' message

0

1

Article ID: KB37230 KB Last Updated: 06 Jul 2021Version: 1.0
Summary:

When trying to commit a new configuration, the following message is seen:

Please check space available in /var/.

The article will explain how to identify the space available in /var/ directory that might forbid any configuration commit.

Symptoms:

The below error message is seen while trying to commit a new configuration:

{MASTER}[edit]
root@ptx-re0# commit
re0:
error: Could not open configuration database (juniper.data+). Please check space available in /var/. To clear up space run the command 'request system storage cleanup'


Also seen in the log message:

Apr 21 11:44:29  ptx-re0 mgd[2723]: LIBJNX_FILE_GENERIC_ERROR: System error: Failed to copy /var/rundb/juniper.db to /var/rundb/juniper.data+ -> No space left on device
Cause:

This issue is due to the creation of a large number of subdirectories by an internal app. As a result, over a period of time, more than a million subdirectories were created.

Solution:

First, use 'request system storage cleanup'. Then commit again. If the issue still exists, then thate the steps below.

  1. Check the IUse% value for the var directory

    user@router>start shell user root

    [vrf:none] user@router:~# df -hi


    Filesystem     Inodes IUsed IFree IUse% Mounted on
    /dev/sda6         64K    26   64K    1% /data/config
    /dev/sda1           0     0     0     - /boot
    /dev/sda2        2.0M  1.3K  2.0M    1% /soft
    /dev/sda5        192K  3.5K  189K    2% /data
    /dev/sda7         9.2M   9.2M     0    100% /var.   <-- 100%
    /dev/loop0       960K    13  960K    1% /data/var/external
    devtmpfs         7.8M  1.2K  7.8M    1% /dev
    uswitchd          476   476     0  100% /
    /tmp             7.8M    90  7.8M    1% /tmp
    run              7.8M  2.8K  7.8M    1% /run
    tmpfs            7.8M  1.3K  7.8M    1% /dev/shm
    tmpfs            7.8M    13  7.8M    1% /sys/fs/cgroup
    tmpfs            7.8M     5  7.8M    1% /run/user/0
  2. Check the No. of the free nodes included:

    [vrf:none] user@router:~# tune2fs -l /dev/sda7 | grep -i inode
    Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize

    Inode count:              9576448
    Free inodes:                 0        <-- no free inodes


    This is due to the large amount of traces created. In the current example, it is due to the sysmanctl app.

    vrf:none] user@router:/var/log/traces# ls -lrt . |grep sysman |wc -l
    4580
    [vrf:none] user@router
    :/var/log/traces# date  
    Thu Nov 12 01:14:17 PST 2020

    [vrf:none] user@router
    :/var/log/traces# ls -lrt . |grep sysman |wc -l
    4906
    [vrf:none] user@router
    :/var/log/traces# date  
    Thu Nov 12 01:18:56 PST 2020

    Approx. 1 subdirectory/sec * 60* 24* 24 = 86400 ~= 100 k subdirectories per day created by sysmanctl with new pids.



    As a workaround, use the 'clear trace' command to clear old traces.

    Starting with release  20.3R2-EVO and higher, a fix is included to hard limit the number of subdirectories created by an app inside /var/log/traces.
Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search