EX2300 and EX3400 switches are equipped with a 2GB flash disk whose disk size is sufficient to be able to store release images, system files, and logs. However, when storage space becomes more heavily utilized over time, there may not be enough disk space available to complete a software update.
This article is written to provide guidance on how to manage storage space and potentially overcome any obstacles tied to space that are encountered in a software update.
If for any reason the information in this article does not address the issues tied to disk space that are encountered during a software update, contact the Support for further assistance.
If an issue with available storage space is encountered during a software update, an error will be returned by the system indicating lack of space. The specific error returned will differ slightly depending on how the switch is deployed, either as a standalone device or as part of a Virtual Chassis.
For a system operating as a standalone device, the following error will be seen:
root@juniper> request system software add /var/tmp/junos-arm-32-18.4R1.8.tgz no-copy no-validate
reboot
Removing /var/log/dcd.0.gz
Removing /var/log/wtmp.0.gz
Removing /var/log/install.0.gz
Removing /var/log/chassisd.1.gz
Removing /var/log/chassisd.3.gz
Removing /var/log/dhcp_logfile.0.gz
Removing /var/log/chassisd.0.gz
Removing /var/log/chassisd.2.gz
Removing /var/log/chassisd.8.gz
Removing /var/log/chassisd.7.gz
Removing /var/log/chassisd.6.gz
Removing /var/log/chassisd.5.gz
Removing /var/log/chassisd.4.gz
Removing /var/preserve/jdhcp_client_data
Removing /var/preserve/jdhcp_client_data_bkp
Error: not enough space to unpack /var/tmp/junos-arm-32-18.4R1.8.tgz
ERROR: Either use 'force' or consider deleting snapshots using 'request system snapshot
delete <snap>'
root@juniper> show system storage
fpc0:
--------------------------------------------------------------------------
Filesystem Size Used Avail Capacity Mounted on
/dev/gpt/junos 1.3G 1.1G 139M 89% /.mount <-- Not enough space
tmpfs 651M 48K 651M 0% /.mount/tmp
tmpfs 324M 436K 324M 0% /.mount/mfs
As seen above, only 139MB is available for use. This is an insufficient amount of space.
For a system operating in a Virtual Chassis, the following error would be returned when the software update is initiated:
root@juniper> request system software add /var/tmp/junos-arm-32-18.1R3-S5.3.tgz unlink reboot force
ERROR: insufficient space for /packages/db/junos-arm-32-18.1R3-S5.3/contents/junos-runtime.tgz
ERROR: insufficient space
ERROR: Failed to add junos-arm-32-18.1R3-S5.3.tgz
And, as in the previous example, the available storage seen from show system storage member <x> is insufficient to handle the update:
root@juniper> show system storage member 3
fpc3:
--------------------------------------------------------------------------
Filesystem Size Used Avail Capacity Mounted on
/dev/gpt/junos 1.3G 1.1G 139M 89% /.mount <-- Not enough space
tmpfs 716M 48K 716M 0% /.mount/tmp
tmpfs 324M 396K 324M 0% /.mount/mfs
The 2GB of total disk space in EX2300 and EX3400 switches is broken down into two partitions. The first partition, which is used by Junos OS, is allocated 1.3GB to 1.5 GB of space. This is the volume seen under /dev/gpt/junos/ above. The second partition that uses the remainder of the disk is tied to the Operation, Administration and Management (OAM) volume. This is used for the storage of recovery snapshots. Recovery snapshots contain full copies of the software packages and configuration that the system is running to be used in the event of a failure.
The Junos OS install package sizes are typically ~300MB, which is the case for the image used in this example (junos-arm-32-18.4R1.8.tgz). So, the 139MB that is available in the system above would be insufficient. However, even if the system had that much space open, the update would still fail. The failure would be tied to how the software installation process works.
The installation process takes the following actions. First, the new software image is copied to the Junos OS volume on the switch. Once the image is copied, the system will check to see if there is enough space to unpack the compressed image. To do so, the switch will unzip the image and the packages found within the .tgz file. Before unzipping, it looks for free space, defined as 3X the package size for the operation to complete. This means that if the image size is 300MB, the update will fail unless there is more than 900MB of available space.
As discussed above, the size of Junos OS volume in EX2300 and EX3400 switches may not be sufficient, given the need for 3X the package size in order to complete the update process. A check of show system storage in a standalone switch or show system storage member <X> via the CLI prior to beginning the update process will give an indication if the update will be successful.
If there is insufficient disk space, Juniper has developed a solution that uses two independent software packages to handle the update. An “os-package” and a “package-hooks-ex” package to help manage disk space and complete the software update. These packages are available at the Juniper Software Download page.
EX2300-EX3400-os-package
EX2300-EX3400-package-hooks-ex
Juniper recommends using these packages to reclaim disk space if there is insufficient space to perform the software update.
Steps:
1. Run the
request system storage cleanup command. This will remove any unnecessary files but keep everything under
/root intact:
root@juniper> request system storage cleanup
2. Run the
request system snapshot delete snap* command. This will delete any existing recovery snapshots that are stored on the system.
root@juniper> request system snapshot delete snap*
Note: If you are unable to delete the recovery.ufs snapshot, please follow the steps under KB32170 - Recovery and Non-recovery snapshots on Junos devices.
3. Copy
os-package and
package-hooks-ex to
/var/tmp.
4. Install the two packages by using the request system software add command:
root@juniper> request system software add /var/tmp/os-package.tgz
root@juniper> request system software add /var/tmp/package-hooks-ex.tgz
5. Install the
Junos OS release software by adding options
no-copy,
unlink, and
force under the
request system software add command:
root@juniper> request system software add /var/tmp/junos-arm-32-18.4R1.8.tgz force unlink no-copy
Note: The unlink option is available starting Junos OS Release 18.1R2, please exclude if running earlier versions. Unlink option removes the .tgz file immediately after the unpack so that the image size calculation takes into account the freed up space tied to the image deletion. This effectively reduces the amount of space needed by the process to 2X instead of 3X. Note that there are some occasions where the add will fail. Try it a second time before moving on.
In Virtual Chassis scenarios where the upgrade fails on the Backup RE or Linecards with log "ERROR: No packages added", option no-copy may need to be excluded.
6. Reboot the system to complete the Junos OS release SW installation process.
After the upgrade is complete, remove the two packages and reboot the system. The show version command will provide information about install packages as shown in the below example.
- os-package
-
package hooks
root@juniper> show version
fpc0:
--------------------------------------------------------------------------
Hostname: d11-12
Model: ex3400-48t
Junos: 18.2R3-S2.7
JUNOS OS Kernel 32-bit [20191022.14c2ad5_builder_stable_11]
JUNOS OS libs [20191022.14c2ad5_builder_stable_11]
JUNOS OS runtime [20191022.14c2ad5_builder_stable_11]
JUNOS OS time zone information [20191022.14c2ad5_builder_stable_11]
JUNOS py extensions [20191029.212835_builder_junos_182_r3_s2]
JUNOS py base [20191029.212835_builder_junos_182_r3_s2]
JUNOS network stack and utilities [20191029.212835_builder_junos_182_r3_s2]
JUNOS libs [20191029.212835_builder_junos_182_r3_s2]
JUNOS runtime [20191029.212835_builder_junos_182_r3_s2]
JUNOS package hooks ex [20191022.160523_sjg_dev_common]
JUNOS OS package [20191113.002318_sjg_stable_11]
JUNOS OS crypto [20191022.14c2ad5_builder_stable_11]
JUNOS Web Management Platform Package [20191029.212835_builder_junos_182_r3_s2]
Delete the two packages by using the request system software delete command:
root@juniper> request system software delete os-package
root@juniper> request system software delete package-hooks-platform
Now reboot the system to complete the uninstallation process for the packages:
root@juniper> request system reboot all-members
Format Install
If the above methods do not work, you can recover the switch by using the method identified in KB31265 - How to format install on EX2300 / EX3400. Note that this procedure will format the switch and all configurations will be lost.
Notes:
- NSSU will not work with these options.
- It is highly recommended to take the necessary recovery snapshots after the upgrade activity completes.
2021-03-11: Removed former step 3 as per PR1572323 conversation:
os-package and package-hooks-ex should do the same without the need to be root
2020-10-02: Non-technical edits.
2020-08-18: Added "Note: If you are unable to delete the recovery.ufs snapshot, follow the steps under KB32170."
2020-02-10: Added the two packages and their download link for seamless software upgrades
2019-05-14: Removed old commands and added new data
2019-05-03: Removed sentences that might cause problems to device
2018-11-02: Minor edit. Non-technical
2018-10-29: Updated with 'force unlink'
2018-02-01: Added 'Please raise a JTAC case if the problem persists'
2017-08-07: Added example for checking snapshots in the Solution section.
2017-05-19: Added information about 3x image size requirement for the upgrade and not 1.5 times. Also added Junos 15.1X53-D56 option to use "force" command while upgrading.
2017-03-21: Removed references to OCCAM.