Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[EX] Not enough storage while upgrading Junos EX2300 and EX3400

0

1

Article ID: KB31198 KB Last Updated: 10 Feb 2020Version: 11.0
Summary:

EX2300 and EX3400 switches are equipped with a 2GB flash disk whose disk size is sufficient to be able to store release images, system files, and logs. However, when storage space becomes more heavily utilized over time, there may not be enough disk space available to complete a software update.

This article is written to provide guidance on how to manage storage space and potentially overcome any obstacles tied to space that are encountered in a software update.

If for any reason the information in this article does not address the issues tied to disk space that are encountered during a software update, contact the Support for further assistance.

 

Symptoms:

If an issue with available storage space is encountered during a software update, an error will be returned by the system indicating lack of space. The specific error returned will differ slightly depending on how the switch is deployed, either as a standalone device or as part of a Virtual Chassis.

For a system operating as a standalone device, the following error will be seen:

root@juniper> request system software add /var/tmp/junos-arm-32-18.4R1.8.tgz no-copy no-validate
reboot
Removing /var/log/dcd.0.gz
Removing /var/log/wtmp.0.gz
Removing /var/log/install.0.gz
Removing /var/log/chassisd.1.gz
Removing /var/log/chassisd.3.gz
Removing /var/log/dhcp_logfile.0.gz
Removing /var/log/chassisd.0.gz
Removing /var/log/chassisd.2.gz
Removing /var/log/chassisd.8.gz
Removing /var/log/chassisd.7.gz
Removing /var/log/chassisd.6.gz
Removing /var/log/chassisd.5.gz
Removing /var/log/chassisd.4.gz
Removing /var/preserve/jdhcp_client_data
Removing /var/preserve/jdhcp_client_data_bkp
Error: not enough space to unpack /var/tmp/junos-arm-32-18.4R1.8.tgz
ERROR: Either use 'force' or consider deleting snapshots using 'request system snapshot
delete <snap>'

root@juniper> show system storage
fpc0:
--------------------------------------------------------------------------
Filesystem      Size  Used  Avail  Capacity  Mounted on
/dev/gpt/junos  1.3G  1.1G  139M   89%       /.mount <-- Not enough space
tmpfs           651M  48K   651M   0%        /.mount/tmp
tmpfs           324M  436K  324M   0%        /.mount/mfs

As seen above, only 139MB is available for use.  This is an insufficient amount of space.

For a system operating in a Virtual Chassis, the following error would be returned when the software update is initiated:

root@juniper> request system software add /var/tmp/junos-arm-32-18.1R3-S5.3.tgz unlink reboot force 
ERROR: insufficient space for /packages/db/junos-arm-32-18.1R3-S5.3/contents/junos-runtime.tgz
ERROR: insufficient space
ERROR: Failed to add junos-arm-32-18.1R3-S5.3.tgz

And, as in the previous example, the available storage seen from show system storage member <x> is insufficient to handle the update:

root@juniper> show system storage member 3   
fpc3:
--------------------------------------------------------------------------
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/gpt/junos          1.3G       1.1G       139M       89%  /.mount <-- Not enough space
tmpfs                   716M        48K       716M        0%  /.mount/tmp
tmpfs                   324M       396K       324M        0%  /.mount/mfs

 

Cause:

The 2GB of total disk space in EX2300 and EX3400 switches is broken down into two partitions. The first partition, which is used by Junos OS, is allocated 1.3GB to 1.5 GB of space. This is the volume seen under /dev/gpt/junos/ above. The second partition that uses the remainder of the disk is tied to the Operation, Administration and Management (OAM) volume. This is used for the storage of recovery snapshots. Recovery snapshots contain full copies of the software packages and configuration that the system is running to be used in the event of a failure.

The Junos OS install package sizes are typically ~300MB, which is the case for the image used in this example (junos-arm-32-18.4R1.8.tgz). So, the 139MB that is available in the system above would be insufficient. However, even if the system had that much space open, the update would still fail. The failure would be tied to how the software installation process works.

The installation process takes the following actions. First, the new software image is copied to the Junos OS volume on the switch. Once the image is copied, the system will check to see if there is enough space to unpack the compressed image. To do so, the switch will unzip the image and the packages found within the .tgz file. Before unzipping, it looks for free space, defined as 3X the package size for the operation to complete. This means that if the image size is 300MB, the update will fail unless there is more than 900MB of available space.

 

Solution:

As discussed above, the size of Junos OS volume in EX2300 and EX3400 switches may not be sufficient, given the need for 3X the package size in order to complete the update process. A check of show system storage in a standalone switch or show system storage member <X> via the CLI prior to beginning the update process will give an indication if the update will be successful.

If there is insufficient disk space, Juniper has developed a solution that uses two independent software packages to handle the update. An “os-package” and a “package-hooks-ex” package to help manage disk space and complete the software update. These packages are available at the Juniper Software Download page.

EX2300-EX3400-os-package

EX2300-EX3400-package-hooks-ex

Juniper recommends using these packages to reclaim disk space if there is insufficient space to perform the software update.

Using the two-package method to upgrade the switch software will have slightly different steps depending on the release that is running on the switch.

Method 1

Use this method if you are currently running image 15.1X53-Dxx. For installed images 18.1R2 or later, refer to Method 2.

  1. Run the request system storage cleanup command. This will remove unnecessary files but keep everything under /root intact.
root@juniper> request system storage cleanup
  1. Run the request system snapshot delete snap* command. This will delete any existing recovery snapshot that stored on the system.
root@juniper> request system snapshot delete snap*
  1. Copy os-package and package-hooks-ex to /var/tmp.
  2. Install the two Junos OS packages named os-package and package-hooks-ex.

root@juniper> request system software add /var/tmp/os-package.tgz
root@juniper> request system software add /var/tmp/package-hooks-ex.tgz
  1. Install the Junos OS release software by using the no-copy and force options under request system software add:
root@juniper> request system software add /var/tmp/junos-arm-32-18.4R1.8.tgz no-copy force

Note that there are some occasions where the add will fail.  Try it a second time before moving on.

  1. Reboot the system to complete the Junos OS installation process.

Method 2

If you are currently running image 18.1R2 or later, follow these steps to update the software:

  1. Run the request system storage cleanup command. This will remove any unnecessary files but keep everything under /root intact:
root@juniper> request system storage cleanup
  1.  Run the request system snapshot delete snap* command. This will delete any existing recovery snapshots that are stored on the system.
root@juniper> request system snapshot delete snap*
  1. Copy os-package and package-hooks-ex to /var/tmp.
  2. Install the two packages by using the request system software add" command:

root@juniper> request system software add /var/tmp/os-package.tgz
root@juniper> request system software add /var/tmp/package-hooks-ex.tgz
  1. Install the Junos OS release software by using the no-copy, unlink, and force options under the request system software add command:
root@juniper> request system software add /var/tmp/junos-arm-32-18.4R1.8.tgz force unlink no-copy

The unlink option is available in Junos OS Release 18.1R2. It removes the .tgz file immediately after the unpack so that the image size calculation takes into account the freed up space tied to the image deletion. This effectively reduces the amount of space needed by the process to 2X instead of 3X. Note that there are some occasions where the add will fail. Try it a second time before moving on.

  1. Reboot the system to complete the Junos OS release SW installation process.

After the upgrade is complete, remove the two packages and reboot the system. The show version command will provide information about install packages as shown in the below example.

  1. os-package
  2. package hooks

root@juniper> show version                             
fpc0:
--------------------------------------------------------------------------
Hostname: d11-12
Model: ex3400-48t
Junos: 18.2R3-S2.7
JUNOS OS Kernel 32-bit  [20191022.14c2ad5_builder_stable_11]
JUNOS OS libs [20191022.14c2ad5_builder_stable_11]
JUNOS OS runtime [20191022.14c2ad5_builder_stable_11]
JUNOS OS time zone information [20191022.14c2ad5_builder_stable_11]
JUNOS py extensions [20191029.212835_builder_junos_182_r3_s2]
JUNOS py base [20191029.212835_builder_junos_182_r3_s2]
JUNOS network stack and utilities [20191029.212835_builder_junos_182_r3_s2]
JUNOS libs [20191029.212835_builder_junos_182_r3_s2]
JUNOS runtime [20191029.212835_builder_junos_182_r3_s2]
JUNOS package hooks ex [20191022.160523_sjg_dev_common]
JUNOS OS package [20191113.002318_sjg_stable_11]
JUNOS OS crypto [20191022.14c2ad5_builder_stable_11]
JUNOS Web Management Platform Package [20191029.212835_builder_junos_182_r3_s2]

Delete the two packages by using the request system software delete command:

root@juniper> request system software delete os-package   
root@juniper> request system software delete package-hooks-platform

Now reboot the system to complete the uninstallation process for the packages:

root@juniper> request system reboot all-members 

Format Install

If the above methods do not work, you can recover the switch by using the method identified in KB31265 - How to format install on EX2300 / EX3400. Note that this procedure will format the switch and all configurations will be lost.

Note

NSSU will not work with these options.

It is highly recommended to take the necessary recovery snapshots after the upgrade activity completes.

 

Modification History:

2020-02-10: Added the two packages and their download link for seamless software upgrades

2019-05-14: Removed old commands and added new data

2019-05-03: Removed sentences that might cause problems to device

2018-11-02: Minor edit. Non-technical

2018-10-29: Updated with 'force unlink'

2018-02-01: Added 'Please raise a JTAC case if the problem persists'

2017-08-07: Added example for checking snapshots in the Solution section.

2017-05-19: Added information about 3x image size requirement for the upgrade and not 1.5 times. Also added Junos 15.1X53-D56 option to use "force" command while upgrading.

2017-03-21: Removed references to OCCAM.

 

Related Links

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search