Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[MX] Managing file system storage to prepare for Junos OS upgrade/downgrade

0

0

Article ID: KB36415 KB Last Updated: 08 Mar 2021Version: 1.0
Summary:
 

This article explains how to manage file system storage when you are preparing a device for Junos OS upgrade or downgrade. The recommendation is to monitor the show system snapshot output as well and delete unnecessary old snapshots while attempting to free up space in preparation for Junos OS upgrade/downgrade.

 

Symptoms:
 

Note: Minimum free space recommended in order to successfully complete a Junos OS upgrade or downgrade is two to three times the Junos OS package file size.

In this example, we want to achieve a minimum of 5G ~ 6G free space before upgrading JunosOS.

After running the request system storage cleanup dry-run command and deleting the unnecessary files, only 4.2G of free space is achieved. We need approximately 5G to 6G if we want to perform a Junos OS upgrade.

{MASTER}
MX> show system storage | match "/var/tmp"                  
/var/jail/tmp            19G        13G       4.2G       76%  /.mount/packages/mnt/jweb-ex32-3b9ae056/jail/var/tmp
/var/tmp                 19G        13G       4.2G       76%  /.mount/packages/mnt/jweb-ex32-3b9ae056/jail/var/tmp/uploads

{MASTER}
MX> start shell command "df -haT | egrep 'File|ufs'" | no-more    
Filesystem                                   Type           Size    Used   Avail Capacity  Mounted on
/dev/gpt/junos                               ufs             19G     13G    4.2G    76%    /.mount
 
show system storage | no-more 
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/md0.uzip            21M        21M         0B      100%  /
devfs                   1.0K       1.0K         0B      100%  /dev
/dev/gpt/junos           19G        13G       4.2G       76%  /.mount   >>>> To check free space
devfs                   1.0K       1.0K         0B      100%  /.mount/dev
/dev/md1.uzip            23M        23M         0B      100%  /.mount/packages/mnt/os-libs-10
/dev/md2.uzip            21M        21M         0B      100%  /.mount/packages/mnt/os-runtime
procfs                  4.0K       4.0K         0B      100%  /.mount/proc
tmpfs                    32G        28K        32G        0%  /.mount/tmp
/dev/md3.uzip           1.6M       1.6M         0B      100%  /.mount/packages/mnt/os-zoneinfo64-fac8062d
/dev/md4.uzip            18M        18M         0B      100%  /.mount/packages/mnt/os-libs-compat32-10
/dev/md5.uzip           538K       538K         0B      100%  /.mount/packages/mnt/os-compat32
/dev/md6.uzip           6.5M       6.5M         0B      100%  /.mount/packages/mnt/py-extensions32-06a9695a
/dev/md7.uzip            14M        14M         0B      100%  /.mount/packages/mnt/py-base32-e1f92a97
/dev/md8.uzip           7.9M       7.9M         0B      100%  /.mount/packages/mnt/os-crypto
/dev/md9.uzip            16M        16M         0B      100%  /.mount/packages/mnt/junos-net
/dev/md10.uzip          7.0M       7.0M         0B      100%  /.mount/packages/mnt/junos-libs
/dev/md11.uzip           50M        50M         0B      100%  /.mount/packages/mnt/junos-libs-compat32
/dev/md12.uzip          724M       724M         0B      100%  /.mount/packages/mnt/junos-runtime32-3a309ce1
/packages/mnt/junos-libs-compat32/usr/lib32        50M        50M         0B  100%  /.mount/packages/mnt/junos-runtime32-3a309ce1/web-api/libs/junos
/packages/mnt/os-libs-compat32-10/usr/lib32        18M        18M         0B  100%  /.mount/packages/mnt/junos-runtime32-3a309ce1/web-api/libs/os
/packages/mnt/os-compat32/libexec       538K       538K         0B  100%  /.mount/packages/mnt/junos-runtime32-3a309ce1/web-api/libexec
/var/jails/rest-api        19G        13G       4.2G     76%  /.mount/packages/mnt/junos-runtime32-3a309ce1/web-api/var
/dev/md13                23M        23M         0B      100%  /.mount/packages/mnt/jweb-ex32-3b9ae056
/var/jail/etc            19G        13G       4.2G       76%  /.mount/packages/mnt/jweb-ex32-3b9ae056/jail/var/etc
/var/jail/run            19G        13G       4.2G       76%  /.mount/packages/mnt/jweb-ex32-3b9ae056/jail/var/run
/var/jail/tmp            19G        13G       4.2G       76%  /.mount/packages/mnt/jweb-ex32-3b9ae056/jail/var/tmp
/var/tmp                 19G        13G       4.2G       76%  /.mount/packages/mnt/jweb-ex32-3b9ae056/jail/var/tmp/uploads
devfs                   1.0K       1.0K         0B      100%  /.mount/packages/mnt/jweb-ex32-3b9ae056/jail/dev
/dev/md14                17M        17M         0B      100%  /.mount/packages/mnt/jweb-ex32-3b9ae056/jail/rt
/var/jail/jweb-app        19G        13G       4.2G      76%  /.mount/packages/mnt/jweb-ex32-3b9ae056/jail/var/jweb-app
/dev/md15.uzip          1.6M       1.6M         0B      100%  /.mount/packages/mnt/junos-libs-compat32-mx
/dev/md16.uzip          343M       343M         0B      100%  /.mount/packages/mnt/junos-runtime-mx32-1a627ba2
/dev/md17.uzip           51M        51M         0B      100%  /.mount/packages/mnt/junos-platform
/dev/md18.uzip           32M        32M         0B      100%  /.mount/packages/mnt/junos-modules64-1d80b057
/dev/md19.uzip           23M        23M         0B      100%  /.mount/packages/mnt/junos-modules-mx64-8e645a57
/dev/md20.uzip          436K       436K         0B      100%  /.mount/packages/mnt/junos-libs-mx
/dev/md21.uzip          8.5M       8.5M         0B      100%  /.mount/packages/mnt/junos-dp-crypto-support32-eac069ba
/dev/md22.uzip           35M        35M         0B      100%  /.mount/packages/mnt/junos-dp-crypto-support-mtx32-73edb6ed
/dev/md23.uzip           89M        89M         0B      100%  /.mount/packages/mnt/junos-daemons64-a06b58de
/dev/md24.uzip           25M        25M         0B      100%  /.mount/packages/mnt/junos-daemons-mx64-ceb022ce
/dev/md25.uzip           15M        15M         0B      100%  /.mount/packages/mnt/jsdn32-78b6749a
/dev/md26.uzip          7.8M       7.8M         0B      100%  /.mount/packages/mnt/jsd32-ee7539e9
/dev/md27.uzip          545M       545M         0B      100%  /.mount/packages/mnt/jpfe-wrlinux32-7082852a
/dev/md28.uzip          605M       605M         0B      100%  /.mount/packages/mnt/jpfe-X96032-56cbb744
/dev/md29.uzip           58M        58M         0B      100%  /.mount/packages/mnt/jpfe-common32-b5e1ff57
/dev/md30.uzip           13M        13M         0B      100%  /.mount/packages/mnt/jdocs32-af44a5b2
/dev/md31.uzip           14M        14M         0B      100%  /.mount/packages/mnt/fips-mode32-b5663a69
tmpfs                   5.3G        48M       5.3G        1%  /.mount/mfs
/dev/md32.uzip          669M       669M         0B      100%  /.mount/packages/mnt/jpfe-X32-d5ea4817
junosprocfs             4.0K       4.0K         0B      100%  /.mount/junosproc

 

Cause:
 

The request system storage cleanup dry-run command does not list "used snapshot storage." It lists all core-dump files and all files in /var/tmp, /var/log, and old/current Junos software packages.

{MASTER}
MX> request system storage cleanup dry-run 
Size   Date         Name
11B Jan 31  2020 /var/jail/tmp/alarmd.ts
51.8K Oct 22 02:24 /var/log/apsd.0.gz
56.0K Aug  4 02:34 /var/log/apsd.1.gz
264.2K Oct 30 20:27 /var/log/chassisd.0.gz
262.9K Jul 24 10:36 /var/log/chassisd.1.gz
262.5K May  1  2020 /var/log/chassisd.2.gz
259.4K Feb  9  2020 /var/log/chassisd.3.gz
258.9K Oct 22  2019 /var/log/chassisd.4.gz
257.3K Jun 23  2019 /var/log/chassisd.5.gz
251.5K Feb 20  2019 /var/log/chassisd.6.gz
269.9K Oct 29  2018 /var/log/chassisd.7.gz
265.7K Aug 11  2018 /var/log/chassisd.8.gz
46.4K Jun 16 05:11 /var/log/dcd.0.gz
45.6K Oct  8  2019 /var/log/dcd.1.gz
34.7K May  5  2020 /var/log/dcd_commit_check.0.gz
35.3K Nov  7  2019 /var/log/dcd_commit_check.1.gz
..
1027B Jun 20  2018 /var/log/install.0.gz
1002B Feb 12  2018 /var/log/install.1.gz
1053B Oct 11  2017 /var/log/install.2.gz
31.2K Nov 23 04:28 /var/log/interactive-commands.0.gz
59.6K Nov 23 04:15 /var/log/interactive-commands.1.gz
59.5K Nov 23 03:45 /var/log/interactive-commands.2.gz
59.6K Nov 23 03:15 /var/log/interactive-commands.3.gz
59.3K Nov 23 02:45 /var/log/interactive-commands.4.gz
59.4K Nov 23 02:15 /var/log/interactive-commands.5.gz
59.4K Nov 23 01:45 /var/log/interactive-commands.6.gz
59.3K Nov 23 01:15 /var/log/interactive-commands.7.gz
59.8K Nov 23 00:45 /var/log/interactive-commands.8.gz
59.5K Nov 23 00:15 /var/log/interactive-commands.9.gz
10.8K Nov 23 04:27 /var/log/jam_chassisd.0.gz
10.8K Nov 23 04:20 /var/log/jam_chassisd.1.gz
10.8K Nov 23 04:13 /var/log/jam_chassisd.2.gz
10.8K Nov 23 04:06 /var/log/jam_chassisd.3.gz
81.5K Jun 20  2018 /var/log/jam_clksyncd.0.gz
73.3K Jun 28  2018 /var/log/jam_cosd.0.gz
72.6K Jun 20  2018 /var/log/jam_cosd.1.gz
72.4K Jun 20  2018 /var/log/jam_cosd.2.gz
73.0K Mar 15  2018 /var/log/jam_cosd.3.gz
56.0K Jun 20  2018 /var/log/jam_dfwd.0.gz
218.2K Nov 13 04:29 /var/log/mastership.0.gz
218.0K Nov  2 08:58 /var/log/mastership.1.gz
218.0K Oct 22 13:27 /var/log/mastership.2.gz
217.7K Oct 11 17:59 /var/log/mastership.3.gz
217.8K Sep 30 22:26 /var/log/mastership.4.gz
217.8K Sep 20 02:54 /var/log/mastership.5.gz
217.9K Sep  9 07:25 /var/log/mastership.6.gz
218.5K Aug 29 11:53 /var/log/mastership.7.gz
217.7K Aug 18 16:21 /var/log/mastership.8.gz
5306B Nov 23 04:28 /var/log/messages.0.gz
48.3K Nov 23 00:45 /var/log/messages.1.gz
53.7K Nov  7 08:30 /var/log/messages.10.gz
54.0K Nov  7 08:15 /var/log/messages.11.gz
43.4K Nov  6 05:00 /var/log/messages.12.gz
49.7K Nov  5 04:15 /var/log/messages.13.gz
51.2K Nov  3 09:15 /var/log/messages.14.gz
48.9K Nov  1 20:00 /var/log/messages.15.gz
48.9K Oct 31 01:30 /var/log/messages.16.gz
48.7K Oct 29 06:45 /var/log/messages.17.gz
51.2K Oct 27 12:00 /var/log/messages.18.gz
47.6K Oct 25 23:00 /var/log/messages.19.gz
49.9K Nov 21 05:30 /var/log/messages.2.gz
50.1K Nov 19 11:00 /var/log/messages.3.gz
50.8K Nov 17 17:15 /var/log/messages.4.gz
49.0K Nov 16 06:00 /var/log/messages.5.gz
49.6K Nov 14 11:30 /var/log/messages.6.gz
48.6K Nov 12 17:00 /var/log/messages.7.gz
49.8K Nov 10 21:45 /var/log/messages.8.gz
49.1K Nov  9 03:15 /var/log/messages.9.gz
...
1999.1M Oct 26  2018 /var/tmp/junos-install-mx-x86-64-16.1R6-S4.2.tgz     >>Old junos
2684.3M Nov 19 23:04 /var/tmp/junos-install-mx-x86-64-18.4R1-S7.1.tgz     >>>New junos

Note: Ensure that you do not delete current and new Junos OS files during the Junos OS upgrade procedure to free up space. The Junos code (16.1R6-S4.2) can be deleted after successfully upgrading the device to the new Junos OS (18.4R1-S7.1). All other files can be deleted to free space.

 

Solution:
 

To increase free up space, we can check the backup snapshot memory usage, and then delete unnecessary snapshots.

Note: Ensure that you do not delete the current Junos OS snapshot and the recovery snapshot.

  1. Check the space consumed by non-recovery snapshots first, and then recovery snapshots. 

MX> show system snapshot 

Non-recovery snapshots:
Snapshot snap.20200620.12833:
Location: /packages/sets/snap.20200620.12833
Creation date: Jun 20 1:28:33 2020
Junos version: 15.1FX.Y

Snapshot snap.20200712.043214:
Location: /packages/sets/snap.20200712.043214
Creation date: Jul 12 04:32:14 2020
Junos version: 16.1RZ-SX.Y

Snapshot snap.20200820.021921:
Location: /packages/sets/snap.20200820.021921
Creation date: Aug 20 02:19:21 2020
Junos version: 16.1RZ-SX.Y

Snapshot snap.20201019.044230:
Location: /packages/sets/snap.20201019.044230
Creation date: Oct 19 04:42:31 2020
Junos version: 16.1RZ-SX.Y

Snapshot snap.20201230.131430:
Location: /packages/sets/snap.20201230.131430
Creation date: Dec 30 13:14:30 2020
Junos version: 16.1RZ-SX.y

Total non-recovery snapshots: 5

Recovery Snapshots:
Snapshots available on the OAM volume:
recovery.ufs
Date created: Sun Jun 20 1:28:33 UTC 2020
cat: /oam/snapshot/VERSION: No such file or directory
Junos version:

Total recovery snapshots: 1
  1. If there are any non-recovery snapshots consuming a lot of space, delete those by using the following command:

{MASTER}
MX> request system snapshot delete snap.20200620.12833                
NOTICE: Snapshot 'snap.20200620.12833' deleted successfully

{MASTER}
MX> request system snapshot delete snap.20200712.043214                
NOTICE: Snapshot 'snap.20200712.043214' deleted successfully

{MASTER}
MX> request system snapshot delete snap.20200820.021921                
NOTICE: Snapshot 'snap.20200820.021921' deleted successfully

{MASTER}
MX> request system snapshot delete snap.20201019.044230  
NOTICE: Snapshot 'snap.20201019.044230' deleted successfully
  1. Now check the space available:

{MASTER}
MX> start shell command "df -haT | egrep 'File|ufs'" | no-more         
Filesystem                                   Type           Size    Used   Avail Capacity  Mounted on
/dev/gpt/junos                               ufs             19G    9.4G    8.2G    53%    /.mount

 

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search