Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[MX] How to repair a corrupted partition in a router running Junos with FreeBSD 10 kernel (Junos 15.1+)

0

0

Article ID: KB32176 KB Last Updated: 04 May 2021Version: 3.0
Summary:

In some cases, a Routing Engine running Junos with FreeBSD 10 kernel (Junos OS 15.1+) may encounter a file system corruption. This can happen in either the primary partition residing in the hard drive (labeled /junos), or in the recovery partition residing in the compact flash (labeled /oam).

This article explains the symptoms and suggests a repair procedure using fsck.

Symptoms:

The corruption in the primary partition can cause the system to reboot and load Junos OS from the OAM partition, or get stuck in a boot loop if the system is unable to load Junos OS from the OAM partition. This is what you can expect if you have a corruption in the main partition:

Main Menu
1. Boot [J]unos volume
2. Boot Junos volume in [S]afe mode
3. [R]eboot
4. [B]oot menu
5. [M]ore options

Choice:
Booting from Junos volume ...
/
Rebooting...

/
Booting from recovery snapshot ...
[...]
--- JUNOS 15.1R5-S4.2 Kernel 64-bit  JNPR-10.3-20170523.350481_build
Note: Junos is currently running in recovery mode on the OAM volume

root@router1-re0:~ #
root@router1-re0:~ # cli
root@router1-re0> show system storage
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/md0.uzip            21M        21M         0B      100%  /
devfs                   1.0K       1.0K         0B      100%  /dev
tmpfs                    14G       360M        14G        2%  /.mount
/dev/gpt/oam            3.8G       853M       2.6G       24%  /mnt
<..>

Corruption in the recovery partition will make it impossible to restore the system in case the primary partition fails, and can also cause a future Junos OS upgrade to fail because it will need the /oam partition. You can check if you have a corrupted /oam partition by using the following CLI command:

user1@router1-re0> show system snapshot     

No non-recovery snapshots available on the Junos volume
Recovery Snapshots:
mount: /dev/gpt/oam: Invalid argument

If you attempt an upgrade with a corrupted /oam partition, you will see the following error:

Verified oam signed by PackageProductionEc_2017
mount: /dev/gpt/oam: R/W mount of /.mount/oam denied. Filesystem is not clean - run fsck.: Operation not permitted
mount: /dev/gpt/oam: R/W mount of /.mount/oam denied. Filesystem is not clean - run fsck.: Operation not permitted
Verified os-compat32 signed by PackageProductionEc_2017
Verified os-crypto signed by PackageProductionEc_2017
Verified os-kernel-prd signed by PackageProductionEc_2017
Verified os-libs-compat32 signed by PackageProductionEc_2017
Verified os-libs signed by PackageProductionEc_2017
Verified os-runtime signed by PackageProductionEc_2017
Verified os-vmguest signed by PackageProductionEc_2017
Verified os-zoneinfo signed by PackageProductionEc_2017
Verified py-base signed by PackageProductionEc_2017
Verified py-extensions signed by PackageProductionEc_2017
Verified vrr-mx signed by PackageProductionEc_2017
ERROR: Failed to add junos-install-mx-x86-64-16.1R4-S4.3.tgz
Solution:

This procedure will let you recover the affected partition by using the fsck tool.

Corruption in Main Partition

  • If the corruption is in the main partition, perform the following:

    • Reboot the RE that is reporting the error.

    • Wait until you see the Boot menu and press the Space bar to stop the Autoboot:

Will try to boot from                                                          
USB Flash Drive                                                                
Compact Flash                                                                  
Solid State Drive 1                                                            
Solid State Drive 2                                                            
Network                                                                        
Trying to boot from Compact Flash
 
Main Menu                
1.  Boot [J]unos volume
2.  Boot Junos volume in [S]afe mode                                  
3.  [R]eboot    
4.  [B]oot menu
5.  [M]ore options                  
Choice: Autoboot in 3 seconds...
  • Press the Space bar and enter 5 to go to the Options menu:                                

Options Menu
1.  Recover [J]unos volume
2.  Recovery mode - [C]LI                  
3.  Check [F]ile system                
4.  Enable [V]erbose boot                  
5.  [B]oot prompt                   
6.  [M]ain menu
  • Enter 3 to choose "Check [F]ile system". You will see the # prompt. Issue the following command:                   

# fsck_ufs /dev/ada0p2    
  • You will see an output similar to this:          

# fsck_ufs /dev/ada0p2
            ** /dev/ada0p2
            ** Last Mounted on /.mount
 ** Phase 1 - Check Blocks and Sizes
 ** Phase 2 - Check Pathnames
 ** Phase 3 - Check Connectivity
 ** Phase 4 - Check Reference Counts
 ** Phase 5 - Check Cyl groups
 SUMMARY BLK COUNT(S) WRONG IN SUPERBLK
 SALVAGE? yes
 
 19141 files, 2921404 used, 2110321 free (1529 frags, 263599 blocks, 0.0% fragmentation)
                            
***** FILE SYSTEM WAS MODIFIED ****                                               
  • Reboot the system with the reboot command.

Corruption in OAM Partition  

  • If the corruption is in the OAM partition, perform the following:

user1@router1-re0> start shell user root
Password:
root@router1-re0:/var/home/user1 #                                  
root@router1-re0:/var/home/user1 # fsck_ufs /dev/ada1p2                       

You will see an output similar to this:   

root@router1-re0:/var/home/user1 # fsck_ufs /dev/ada1p2
** /dev/ada1p2
** Last Mounted on /.mount/oam
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
SUMMARY BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? yes
 
68 files, 455156 used, 533075 free (43 frags, 66629 blocks, 0.0% fragmentation)
                            
***** FILE SYSTEM WAS MODIFIED *****

Corruption in Superblock of Partition

  • In some cases, we may have a file system corruption that affects the superblock of the partition:

root@router1-re0:/var/home/user1 # fsck_ufs /dev/gpt/oam
** /dev/gpt/oam
Cannot find file system superblock
/dev/gpt/oam: CANNOT FIGURE OUT FILE SYSTEM PARTITION
  • In order to recover from this issue, we can use one of two options:

    • Option 1:

user1@router1-re0> request system recover oam-volume
 NOTICE: Recovering the OAM volume ...
 ada1p2 deleted
 ada1 created
 ada1p1 added
 bootcode written to ada1
 ada1p2 added
 /dev/gpt/oam: 3998.0MB (8187840 sectors) block size 32768, fragment size 4096
         using 7 cylinder groups of 626.09MB, 20035 blks, 80256 inodes.
 super-block backups (for fsck_ffs -b #) at:
 192, 1282432, 2564672, 3846912, 5129152, 6411392, 7693632
 Verified oam signed by PackageProductionEc_2017 method ECDSA256+SHA256
 Installing OAM volume contents ...
 The OAM volume is now installed
 NOTICE: Creating a recovery snapshot on the OAM volume ...
 Creating image ...
 Compressing image ...
 Image size is 795MB
 Recovery snapshot created successfully
 NOTICE: OAM volume recovered
  • Option 2:

root@router1-re0:/var/home/user1 # newfs -U /dev/ada1p2

  /dev/ada1p2: 3998.0MB (8187840 sectors) block size 32768, fragment size 4096
          using 7 cylinder groups of 626.09MB, 20035 blks, 80256 inodes.
          with soft updates
  super-block backups (for fsck_ffs -b #) at:
   192, 1282432, 2564672, 3846912, 5129152, 6411392, 7693632

root@router1-re0:/var/home/user1 # fsck_ufs -b 192 /dev/ada1p2

Alternate super block location: 192
** /dev/ada1p2
** Last Mounted on
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
SUMMARY BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? [yn] y
2 files, 2 used, 988229 free (21 frags, 123526 blocks, 0.0% fragmentation)
UPDATE STANDARD SUPERBLOCK? [yn] y
***** FILE SYSTEM IS CLEAN *****    
***** FILE SYSTEM WAS MODIFIED *****
Modification History:
  • 2020-06-18: Article reviewed for accuracy, no changes required; article accurate and valid.

  • 2021-05-04: Minor modification in Solution ( "Corruption in Man Partition") part; the remaining content is valid and accurate.

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search