Support Support Downloads Knowledge Base Juniper Support Portal Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[QFX] QFX5100/QFX5200 Junos recovery

0

0

Article ID: KB36947 KB Last Updated: 24 Jun 2021Version: 3.0
Summary:

This article describes how to recover from a failed software installation of Junos OS or restore the previous working Junos OS configuration if you access the Command Line Interface (CLI).

Symptoms:

The device does not boot up to the CLI due to issues during installation after an upgrade. After a failed installation, the QFX5K devices are left in a state that makes them inaccessible remotely and it needs a new software installation by copying the software from the USB to the device or by using the installation from a USB.

Cause:

There could be various reasons that trigger this issue, such as incompatible configuration for the loaded Junos OS or file-system issues.

Solution:

If the device boots to CLI, a complete reboot is needed, including the Linux host OS. This is done by using request system reboot hypervisor.  Sometimes with a CLI reboot, when pressing "Esc" to get to boot options on the console, the following menu is seen:


In such cases, or when CLI is not accessible, we need to power cycle the QFX to get to the right recovery screen in the screenshots mentioned below.

A recovery image is created automatically on the switches. If a previously-running switch is powered on and unable to boot using a Junos OS image, you can boot the switch by using the recovery Junos OS image from one of the saved images.

Also, once the appropriate recovery image is selected, you get the option to format the disk/configuration, which may help recover from any incompatible configurations that prevent the device from booting to a particular Junos OS image.

For QFX5100/EX4600

boot_menu

For QFX5200

After selecting Linux recovery on QFX5200:

(This is an example of booting up from one of the recovery images without formatting the disk/configuration to show the options and the boot sequence.)

Recovering Linux ...
4..... root=LABEL=ROOT console=ttyS0,9600n8 quiet
i8042: No controller found
First Level Bootstrap using initramfs...
Mounting boot device LABEL=LINUX-BOOT
Unpacking initrd.cpio.gz ....
1033308 blocks
Unmount boot device LABEL=LINUX-BOOT
Starting init
INIT: Mounting VAR to /var
Starting udev
udevd[361]: can not find '/etc/udev/rules.d/81-i40e-udevhelper.rules': No such file or directory

sysvinit-2.88dsf-r500.0.core2_64
running pre rc steps on QFX5_32C_32Q

 Changing boot device enable from  0x00 to 0x00
 0x2C 0x2c
  2 logical volume(s) in volume group "vg0_vjunos" now active
  2 logical volume(s) in volume group "vg0_vjunos" now active

  PV         VG         Fmt  Attr PSize  PFree
  /dev/sda6  vg0_vjunos lvm2 a--   8.86g    0 
  /dev/sdb   vg0_vjunos lvm2 a--  29.82g    0 

  VG         #PV #LV #SN Attr   VSize  VFree
  vg0_vjunos   2   2   0 wz--n- 38.68g    0 

  LV                 VG         Attr     LSize  Pool Origin Data%  Move Log Copy%  Convert
  lv_junos           vg0_vjunos -wi-a--- 30.00g                                           
  lv_var_third_party vg0_vjunos -wi-a---  8.68g                                           



+------------------------------------------+
|   Junos Host applications Recovery       |
+------------------------------------------+

=================================================
Current image  : qfx-5e-18.2X75-D435.20.202104251815
Previous image : qfx-5e-18.2X75-D435.20.202104221228
=================================================

Select an image for recovery or cancel to continue:

Select a recovery image:
1) qfx-5e-18.2X75-D435.20.202104251815
2) qfx-5e-18.2X75-D435.20.202104221228
3) cancel
Pick an option[1-3]:  

Pick an option[1-3]:  1

Do you want to format Junos data disk?[y/n] n  

Do you want to format Junos config disk?[y/n] n

Do you want to reimage linux host?[y/n] n
=================================================
Recovery options selected:
 Recovery image            : qfx-5e-18.2X75-D435.20.202104251815
 Format Junos data disk    : no
 Format Junos config disk  : no
 Reimage linux host        : no
=================================================


Choose one of the following options:
1) Proceed with the recovery.
2) Go back to the recovery menu.
3) Abort the recovery process.

Please confirm[1/2/3]: 1
Setting up Junos host application for upgrade ...

Recovering Junos host application image to: qfx-5e-18.2X75-D435.20.202104251815

Recovery staged.
Rebostopping rsyslogd ... done
Stopping OpenBSD Secure Shell server: sshdno /usr/sbin/sshd found; none killed
Stopping atd: OK
Stopping domain name service: named.
Unmounting cgroups...Done
Stopping system message bus: dbus.
stopping DNS forwarder and DHCP server: dnsmasq... no /usr/bin/dnsmasq found; none killed
Shutting down irqbalance: no irqbalance found; none killed
done
stopping mountd: done
stopping nfsd: done
Stopping random number generator daemon.
stopping rsyslogd ... done
Stopping internet superserver: xinetd.
stopping statd: done
Stopping portmap daemon...
udevd[367]: can not find '/etc/udev/rules.d/81-i40e-udevhelper.rules': No such file or directory

Clearing ebtables rulesets: filter nat broute done. ok
Stopping crond: FAIL
Stopping rpcbind daemon...
not running.
Stopping S.M.A.R.T. daemon: smartd.
Stopping network management services: snmpd snmptrapd libvirtMib_subagent.
 * Stopping virtualization library daemon: libvirtd                      [fail]
Deconfiguring network interfaces... ifdown: interface lo not configured
done.
Stopping tcsd: FAIL
Sending all processes the TERM signal...
Sending all processes the KILL signal...
Unmounting remote filesystems...
Deactivating swap...
Unmounting local filesystems...
Rebooting... RE-FPGA-DRV: reboot notifier called with 0x0001 
RE-FPGA-DRV: Please standby while rebooting. 
.
..
..
3h

   3h                                    

InsydeH2O version : V0018.6U
BIOS Build Date : 10/30/2013

System Memory Speed : 1333 MHz

Processor Type : Intel(R) Xeon(R) CPU  @ 1.80GHz

CPUID : 306A9
Press Esc for boot options
Now booting : SSD0 : TS32GMTS400                     

                             GNU GRUB  version 2.00

 +--------------------------------------------------------------------------+
 |Juniper Linux                                                             | 
 |Juniper Linux  Debug                                                      |
 |Juniper-Linux-Upgrade                                                     |
 |Juniper-Linux-Recovery                                                    |
 |                                                                          |
 |                                                                          |
 |                                                                          |
 |                                                                          |
 |                                                                          |
 |                                                                          |
 |                                                                          |
 |                                                                          | 
 +--------------------------------------------------------------------------+

      Use the ^ and v keys to select which entry is highlighted.      
      Press enter to boot the selected OS, `e' to edit the commands      
      before booting or `c' for a command-line.      
   The highlighted entry will be executed automatically in 0s.
                                                                               
  Booting `Juniper-Linux-Upgrade'

Upgrading Linux ...
i8042: No controller found
First Level Bootstrap using initramfs...
Mounting boot device LABEL=LINUX-BOOT
Unpacking initrd.cpio.gz ....
1033308 blocks
Unmount boot device LABEL=LINUX-BOOT
Starting init
INIT: version 2Mouting VAR to /var
Starting udev
udevd[329]: can not find '/etc/udev/rules.d/81-i40e-udevhelper.rules': No such file or directory

sysvinit-2.88dsf-r500.0.core2_64
running pre rc steps on QFX5_32C_32Q

 Changing boot device enable from  0x00 to 0x00
 0x2C 0x2c
Moving [HOST_COMPAT_VERSION app_install_record.md5 lxc_pkg_install.sh pkg-app-junos.tgz pkg_install.sh platform_mappings qfx-5e-18.2X75-D435.20.tgz version.txt] to /var/sw/packages/applications
  2 logical volume(s) in volume group "vg0_vjunos" now active
udevd[329]: can not find '/etc/udev/rules.d/81-i40e-udevhelper.rules': No such file or directory


  PV         VG         Fmt  Attr PSize  PFree
  /dev/sda6  vg0_vjunos lvm2 a--   8.86g    0 
  /dev/sdb   vg0_vjunos lvm2 a--  29.82g    0 

  VG         #PV #LV #SN Attr   VSize  VFree
  vg0_vjunos   2   2   0 wz--n- 38.68g    0 

  LV                 VG         Attr     LSize  Pool Origin Data%  Move Log Copy%  Convert
  lv_junos           vg0_vjunos -wi-a--- 30.00g                                           
  lv_var_third_party vg0_vjunos -wi-a---  8.68g                                           

Setting up boot params ...
------------------------------------------------
Running Junos application installer
------------------------------------------------

-------------------------------
Installing /var/sw/packages/applications/qfx-5e-18.2X75-D435.20.tgz
-------------------------------

Removing package: qfx-5e-control-plane-1.0-0@x86_64
Removing package: qfx-5e-host-control-1.0-0@x86_64
Removing package: qfx-5e-host-platform-1.0-0@x86_64
pkg_install_rpms: qfx-5e-base-1.0-0-18.2X75-D435.20.x86_64.rpm
pkg_install_rpms: installing LXC supported RPMs
========================================
Installing qfx-5e-host-control-1.0-0-18.2X75-D435.20.x86_64.rpm ...
========================================
Loading cache...
Updating cache...               ######################################## [100%]

Committing transaction...                                                                              
Preparing...                    ######################################## [  0%]
   1:Installing qfx-5e-host-c.. ######################################## [100%]
Output from qfx-5e-host-control-1.0-0@x86_64:
Stopping vehostd: 
Starting vehostd: vehostd startup
touch: cannot touch '/var/lock/subsys/vehostd': No such file or directory


Installing packages (1):
  qfx-5e-host-control-1.0-0@x86_64       

348.3kB of package files are needed. 1.2MB will be used.


========================================
Installing qfx-5e-host-platform-1.0-0-18.2X75-D435.20.x86_64.rpm ...
========================================
Loading cache...
Updating cache...               ######################################## [100%]

Committing transaction...                                                                              
Preparing...                    ######################################## [  0%]
Output from qfx-5e-host-platform-1.0-0@x86_64:
dcpfe0 is not running
   1:Installing qfx-5e-host-p.. ######################################## [100%]


Installing packages (1):
  qfx-5e-host-platform-1.0-0@x86_64      

5.8MB of package files are needed. 22.6MB will be used.


Saving cache...

========================================
Installing qfx-5e-control-plane-1.0-0-18.2X75-D435.20.x86_64.rpm ...
========================================
Loading cache...
Updating cache...               ######################################## [100%]

Committing transaction...                                                                              
Preparing...                    ######################################## [  0%]
   1:Installing qfx-5e-contro.. ######################################## [100%]
Output from qfx-5e-control-plane-1.0-0@x86_64:
-------------------------------
Installing JUNOS image: jinstall-jcp-i386-18.2X75-D435.20.img.gz
-------------------------------
Extracting jinstall-jcp-i386-18.2X75-D435.20.img.gz to /recovery/junos/jinstall-jcp-i386-18.2X75-D435.20-202105060903.img
Prepare host for virtfs...
Integrity check passed for hash-control-plane.md5.


Installing packages (1):
  qfx-5e-control-plane-1.0-0@x86_64      

943.3MB of package files are needed. 954.0MB will be used.


Saving cache...

========================================

 Application installed.
Setting up boot params ...
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
/
---------------------------------------------

 Installation for Junos
Rebooting the system to complete the installation
---------------------------------------------
INIT: Switchingstopping rsyslogd ... done
Stopping OpenBSD Secure Shell server: sshdno /usr/sbin/sshd found; none killed
Stopping atd: OK
Stopping domain name service: named.
Unmounting cgroups...Done
Stopping system message bus: dbus.
stopping DNS forwarder and DHCP server: dnsmasq... no /usr/bin/dnsmasq found; none killed
Shutting down irqbalance: no irqbalance found; none killed
done
stopping mountd: done
stopping nfsd: done
Stopping random number generator daemon.
stopping rsyslogd ... done
Stopping internet superserver: xinetd.
stopping statd: done
Stopping portmap daemon...
udevd[329]: can not find '/etc/udev/rules.d/81-i40e-udevhelper.rules': No such file or directory

Clearing ebtables rulesets: filter nat broute done. ok
Stopping crond: FAIL
Stopping rpcbind daemon...
not running.
Stopping S.M.A.R.T. daemon: smartd.
Stopping network management services: snmpd snmptrapd libvirtMib_subagent.
 * Stopping virtualization library daemon: libvirtd                      [fail]
Deconfiguring network interfaces... ifdown: interface lo not configured
done.
Stopping tcsd: FAIL
Sending all processes the TERM signal...
Sending all processes the KILL signal...
Unmounting remote filesystems...
Deactivating swap...
Unmounting local filesystems...
Rebooting... RE-FPGA-DRV: reboot notifier called with 0x0001 
RE-FPGA-DRV: Please standby while rebooting. 
.
..
..

InsydeH2O version : V0018.6U
BIOS Build Date : 10/30/2013

System Memory Speed : 1333 MHz

Processor Type : Intel(R) Xeon(R) CPU  @ 1.80GHz

CPUID : 306A9
Press Esc for boot options
Now booting : SSD0 : TS32GMTS400                     

                             GNU GRUB  version 2.00

 +--------------------------------------------------------------------------+
 |Juniper Linux                                                             | 
 |Juniper Linux  Debug                                                      |
 |Juniper-Linux-Recovery                                                    |
 |                                                                          |
 |                                                                          |
 |                                                                          |
 |                                                                          |
 |                                                                          |
 |                                                                          |
 |                                                                          |
 |                                                                          |
 |                                                                          | 
 +--------------------------------------------------------------------------+

      Use the ^ and v keys to select which entry is highlighted.      
      Press enter to boot the selected OS, `e' to edit the commands      
      before booting or `c' for a command-line.      
                                                                               
                                                                               
Loading Linux ...
i8042: No controller found
First Level Bootstrap using initramfs...
Mounting boot device LABEL=LINUX-BOOT
Unpacking initrd.cpio.gz ....
1033308 blocks
Unmount boot device LABEL=LINUX-BOOT
Starting init
INIT: Mouting VAR to /var
Starting udev
udevd[329]: can not find '/etc/udev/rules.d/81-i40e-udevhelper.rules': No such file or directory

sysvinit-2.88dsf-r500.0.core2_64
running pre rc steps on QFX5_32C_32Q

 Changing boot device enable from  0x00 to 0x00
 0x2C 0x2c
  2 logical volume(s) in volume group "vg0_vjunos" now active
Starting udev

<<<<<OMMITTED PART OF BOOT SEQUENCE FOR BREVITY>>>>>

Removing any system startup links for nsca ...
update-rc.d: /etc/init.d/hostapd exists during rc.d purge (continuing)
 Removing any system startup links for hostapd ...
update-rc.d: /etc/init.d/wdmd exists during rc.d purge (continuing)
 Removing any system startup links for wdmd ...
  /etc/rc0.d/K20wdmd
  /etc/rc1.d/K20wdmd
  /etc/rc2.d/S20wdmd
  /etc/rc3.d/S20wdmd
  /etc/rc4.d/S20wdmd
  /etc/rc5.d/S20wdmd

Wind River Linux 7.0.0.9 localhost console

INIT: Consoles: serial port  
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 639kB/3144576kB available memory

FreeBSD/x86 bootstrap loader, Revision 1.1
(Thu Apr 23 13:03:02 PDT 2020 builder@feyrith.juniper.net)
-

Booting from Junos volume ...
|
/packages/sets/pending/boot/os-kernel/kernel text=0x45d3d8 data=0x84130+0x30d970 syms=[0x8+0x971d0+0x8+0x8359e]

<<<<<OMMITTED PART OF BOOT SEQUENCE FOR BREVITY>>>>>

Creating initial configuration:  ...
mgd: Running FIPS Self-tests
veriexec: no fingerprint for file='/sbin/kats/cannot-exec' fsid=203 fileid=51404 gen=1 uid=0 pid=5459
mgd: FIPS Self-tests Passed
[1] 5580
mgd: commit complete
@ 1620292523 [2021-05-06 14:45:23 IST] mgd done

 Lock Manager
RDM Embedded 7 [04-Aug-2006] http://www.birdstep.com
Copyright (c) 1992-2006 Birdstep Technology, Inc.  All Rights Reserved.

Unix Domain sockets Lock manager
Lock manager 'lockmgr' started successfully.

Database Initialization Utility
RDM Embedded 7 [04-Aug-2006] http://www.birdstep.com
Copyright (c) 1992-2006 Birdstep Technology, Inc.  All Rights Reserved.

Profile database initialized
No core dumps found.
Prefetching /usr/sbin/rpd ...
Prefetching /usr/libexec64/rpd ...
Prefetching /usr/sbin/lacpd ...
Prefetching /usr/sbin/chassisd ...
Starting jlaunchhelperd.
192.168.1.16 root
192.168.1.17 root
Thu May  6 14:45:33 IST 2021 platform_fixups: 6157 6255
Thu May  6 14:45:33 IST 2021 platform_fixups: Give sighup to (pid:6157) parse new file ...
@ 1620292533 [2021-05-06 14:45:33 IST] mountlater start
@ 1620292538 [2021-05-06 14:45:38 IST] mountlater done
Starting cron.

Thu May  6 14:45:39 IST 2021

FreeBSD/amd64 (jtac-qfx5200-32c-32q-r2001) (ttyu0)

login:           labroot
Password:
Last login: Thu May  6 15:05:56 from 10.219.48.52

--- JUNOS 18.2X75-D435.20 Kernel 64-bit  JNPR-11.0-20200423.d6acdc5_buil
{master:0}
labroot@jtac-qfx5200-32c-32q-r2001> 

{master:0}
labroot@jtac-qfx5200-32c-32q-r2001> 

{master:0}
labroot@jtac-qfx5200-32c-32q-r2001> show version 
localre:
--------------------------------------------------------------------------
Hostname: jtac-qfx5200-32c-32q-r2001
Model: qfx5200-32c-32q
Junos: 18.2X75-D435.20
JUNOS OS Kernel 64-bit  [20200423.d6acdc5_builder_stable_11]
JUNOS OS libs [20200423.d6acdc5_builder_stable_11]
JUNOS OS runtime [20200423.d6acdc5_builder_stable_11]
JUNOS OS time zone information [20200423.d6acdc5_builder_stable_11]
JUNOS OS libs compat32 [20200423.d6acdc5_builder_stable_11]
JUNOS OS 32-bit compatibility [20200423.d6acdc5_builder_stable_11]
JUNOS py extensions [20210215.051819_builder_junos_182_x75_d435]
JUNOS py base [20210215.051819_builder_junos_182_x75_d435]
JUNOS OS vmguest [20200423.d6acdc5_builder_stable_11]
JUNOS OS crypto [20200423.d6acdc5_builder_stable_11]
JUNOS network stack and utilities [20210215.051819_builder_junos_182_x75_d435]
JUNOS libs [20210215.051819_builder_junos_182_x75_d435]
JUNOS libs compat32 [20210215.051819_builder_junos_182_x75_d435]
JUNOS runtime [20210215.051819_builder_junos_182_x75_d435]
JUNOS na telemetry [18.2X75-D435.20]
JUNOS Web Management Platform Package [20210215.051819_builder_junos_182_x75_d435]
JUNOS qfx runtime [20210215.051819_builder_junos_182_x75_d435]
JUNOS common platform support [20210215.051819_builder_junos_182_x75_d435]
JUNOS qfx platform support [20210215.051819_builder_junos_182_x75_d435]
JUNOS Openconfig [18.2X75-D435.20]
JUNOS dcp network modules [20210215.051819_builder_junos_182_x75_d435]
JUNOS modules [20210215.051819_builder_junos_182_x75_d435]
JUNOS qfx modules [20210215.051819_builder_junos_182_x75_d435]
JUNOS qfx Data Plane Crypto Support [20210215.051819_builder_junos_182_x75_d435]
JUNOS daemons [20210215.051819_builder_junos_182_x75_d435]
JUNOS qfx daemons [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services URL Filter package [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services TLB Service PIC package [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services Telemetry [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services SSL [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services SOFTWIRE [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services Stateful Firewall [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services RPM [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services PCEF package [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services NAT [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services Mobile Subscriber Service Container package [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services MobileNext Software package [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services Logging Report Framework package [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services LL-PDF Container package [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services Jflow Container package [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services Deep Packet Inspection package [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services IPSec [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services IDS [20210215.051819_builder_junos_182_x75_d435]
JUNOS IDP Services [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services HTTP Content Management package [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services Flowd MS-MPC Software package [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services Crypto [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services Captive Portal and Content Delivery Container package [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services COS [20210215.051819_builder_junos_182_x75_d435]
JUNOS AppId Services [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services Application Level Gateways [20210215.051819_builder_junos_182_x75_d435]
JUNOS Services AACL Container package [20210215.051819_builder_junos_182_x75_d435]
JUNOS SDN Software Suite [20210215.051819_builder_junos_182_x75_d435]
JUNOS Extension Toolkit [20210215.051819_builder_junos_182_x75_d435]
JUNOS Packet Forwarding Engine Support (DC-PFE) [20210215.051819_builder_junos_182_x75_d435]
JUNOS Packet Forwarding Engine Support (M/T Common) [20210215.051819_builder_junos_182_x75_d435]
JUNOS J-Insight [20210215.051819_builder_junos_182_x75_d435]
JUNOS jfirmware [20210215.051819_builder_junos_182_x75_d435]
JUNOS Online Documentation [20210215.051819_builder_junos_182_x75_d435]
JUNOS jail runtime [20200423.d6acdc5_builder_stable_11]
JUNOS FIPS mode utilities [20210215.051819_builder_junos_182_x75_d435]
JUNOS Host Software [3.14.52-rt50-WR7.0.0.9_ovp:3.0.9]
JUNOS Host qfx-5e host-control package [18.2X75-D435.20]
JUNOS Host qfx-5e host-platform package [18.2X75-D435.20]
JUNOS Host qfx-5e control-plane package [18.2X75-D435.20]
JUNOS Host qfx-5e platform package [18.2X75-D435.20]
JUNOS Host qfx-5e base package [18.2X75-D435.20]
JUNOS Host qfx-5e data-plane package [18.2X75-D435.20]
Modification History:

2021-06-10: Added screenshot for the menu seen with CLI reboot and clarified the need to power cycle if you cannot get to the recovery screen or when CLI isn't accessible for recovery.
2021-06-01: Trimmed down the long boot sequence and highlighted some key portions in the boot sequence to watch out for

Related Links

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search