Support Support Downloads Knowledge Base Service Request Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[EX/QFX] How to troubleshoot ‘/var partition usage is high’

0

0

Article ID: KB30860 KB Last Updated: 05 Mar 2019Version: 3.0
Summary:

This article provides steps on how to troubleshoot what is taking up the disk space in an EX switch.

Symptoms:

Running the command ‘show chassis alarms’ on an EX switch returns the following message:

Minor Host 0 /var partition usage is high.

What could be consuming the disk space?

Example:
When checking the device utilization, the following output is returned:

{master:0}pwd

lab@Dominical> show system storage
/dev/da0s3e 123M 82M 31M 72% /var

This output does not reveal what could be consuming the disk space.

Running the command, ’file list /var/’ does not provide enough information:

.snap/
BSD.var.dist
account/
at /
backups/
bin/
crash/
cron/
db /
empty/
etc/
etcroot/
heimdal/
home/
jail/
log/
logical-systems/
mail/
mfs/
msgs/
named/
preserve/
root/
run/
rundb/
rwho/
spool/
sw /
tmp/
transfer/
validate/
yp /

Solution:

Example scenario using Unix shell commands to find out what is taking up space in an EX switch:

  1. Enter the command 'start shell'

    {master:0}
    lab@Dominical> start shell
    %

    By default, Junos will go to the user home directory. Confirm this by typing pwd.

    % pwd
    /var/home/lab

  2. Go to the upper level of the /var/ partition by using the following commands:

    % cd /var/
    % pwd
    /var

     
  3. Check folder utilization:

    % du -cks * | sort -rn
    du: cron/tabs: Permission denied
    du: db/entropy: Permission denied
    du: db/certs/common: Permission denied
    du: db/certs/system-key-pair: Permission denied
    du: db/certs/system-cert: Permission denied
    du: db/dhcp_snoop: Permission denied
    du: heimdal: Permission denied
    du: root/.ssh: Permission denied
    du: run/ppp: Permission denied
    du: spool/opielocks: Permission denied
    407216 total
    293838 tmp
    53964 home
    30430 rundb
    20698 root
    6888 log
    378 db
    332 mfs
    204 run
    204 etc
    120 jail
    100 etcroot
    12 spool
    6 at
    4 transfer
    4 sw
    4 crash
    4 BSD.var.dist
    2 yp
    2 validate
    2 rwho
    2 preserve
    2 named
    2 msgs
    2 mail
    2 logical-systems
    2 empty
    2 cron
    2 bin
    2 backups
    2 account


    In this case, it appears that most of the disk space is being consumed by the tmp folder (a partition of its own) and the home file. 

  4. There is a limitation with the du tool. We cannot tell if it is an actual file or a directory. Use the following command to confirm:

    % file home
    home: directory


    Alternatively, the following command can be used:

    % ls -l | grep home
    drwxr-xr-x 37 root wheel 1024 Sep 5 18:22 home


    The first column shows a lowercase d, which in Unix, indicates it is a directory.

  5. Enter this directory and run the procedure again to find out what is using up the space inside /var/home:

    % cd home/
    % pwd
    /var/home
    % du -cks * | sort -rn
    53962 total
    53802 lab
    46 daltamirano
    16 remote
    14 wheaslip
    12 jrojas
    4 wmoreira

    drwxr-xr-x 3 lab 20 512 Dec 17 09:30 lab


    This reveals that most of the usage comes from the lab directory.

  6. Repeat the procedure once more to see files under the lab directory:

    % cd lab/
    % du -cks * | sort -rn
    53796 total
    28000 foo
    25760 bar
    14 DHCP-CONFIGURATION-MEMO
    8 et inte
    8 JORGE-TEST
    4 h
    2 DHCPclient.PCAP
    0 LC-DHCP-LAB
    0 DHCP.PCAP

    % ls -l | grep foo
    -rw-r--r-- 1 lab field 28655162 Dec 17 09:13 foo
    % ls -l | grep bar
    -rw-r--r-- 1 lab field 26361809 Dec 17 09:13 bar


    The space is being consumed by two files; foo and bar.

  7. Run the files to find out what they are:

    % file foo
    foo: gzip compressed data, from Unix, max compression
    % file bar
    bar: gzip compressed data, from Unix, max compression


    This reveals they are tarballs. In most scenarios /var/ utilization increases either because tarball upgrade packets were copied to a wrong directory, or a packet capture was running for too long.

    % file DHCPclient.PCAP
    DHCPclient.PCAP: tcpdump capture file (little-endian) - version 2.4, capture length 96)

  8. Remove the files:

    % rm foo
    % rm bar
    % ls
    DHCP-CONFIGURATION-MEMO DHCPclient.PCAP LC-DHCP-LAB h
    DHCP.PCAP JORGE-TEST et inte


    Liberating the space in the process:

    /dev/da0s3e 123M 29M 84M 26% /var

    After about 20 minutes the alarm will clear by itself.

  9. Another helpful command is 'request system storage cleanup dry-run’ 
        
    user@host> request system storage cleanup dry-run
    Currently rotating log files, please wait.
    This operation can take up to a minute.

    List of files to delete:

    Size Date Name
    11.4K Mar 8 15:00 /var/log/messages.1.gz
    7245B Feb 5 15:00 /var/log/messages.3.gz
    11.8K Feb 22 13:00 /var/log/messages.2.gz
    3926B Mar 16 13:57 /var/log/messages.0.gz
    3962B Feb 22 12:47 /var/log/sampled.1.gz
    4146B Mar 8 12:20 /var/log/sampled.0.gz
    4708B Dec 21 11:39 /var/log/sampled.2.gz
    7068B Jan 16 18:00 /var/log/messages.4.gz
    13.7K Dec 27 22:00 /var/log/messages.5.gz
    890B Feb 22 17:22 /var/tmp/sampled.pkts
    65.8M Oct 26 09:10 /var/sw/pkg/jinstall-7.4R1.7-export-signed.tgz
    63.1M Oct 26 09:13 /var/sw/pkg/jbundle-7.4R1.7.tgz

        
Important Notes:
  • Unless you are familiar with the Junos file system, or you have been advised to do so by a JTAC engineer; do not attempt to delete files outside the /var/home directory, as it can break your switch.
  • Unless strictly necessary, do not log into the console as ‘root’, because the switch will allow you to delete critical system files.
  • Remember that Unix does not recognize file extensions, so it is common to have a file with no extension.
  • KB22966 - How to resolve the '/var: filesystem full' issue which occurs as a result of the WTMP file not being archived. This article explains a similar condition related to the WTMP file taking up space.
Modification History:

2019-03-05: Minor edit. Non-technical

Related Links

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Security Alerts and Vulnerabilities

Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search