Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

Instructions to Offline/Online MX Fabric Plane and MX SCB (called from Resolution Guide - Troubleshoot Fabric Plane)

2

0

Article ID: KB23067 KB Last Updated: 22 Mar 2021Version: 2.0
Summary:

This article describes the common guidelines to follow when offlining/onlining MX Fabric Planes and MX SCB.

Symptoms:

When troubleshooting Fabric Plane related issues, the offline and online of a Fabric Plane or SCB may be requested in the following KBs:

KB23173 - Resolution Guide - MX - Troubleshoot Fabric Planes in Fault or Check State
KB23068 - MX SCB Troubleshooting Subroutine
This article provides step-by-step instructions.


This MX960 Switch Fabric diagram can be used for reference in this article. For more information on the terminology and other platforms, refer to KB23065 - Understanding MX Fabric Planes.



 
Solution:

Note: It is strongly recommended to perform the following procedures during in a Maintenance Window.

Also, these instructions assume that the MX router is configured with redundant Fabric capacity (MX960 3 SCB's and MX480/MX240 2 SCB's). If redundant fabric planes are not available, then the above guidelines are not recommended on production routers. 

Click on the link to jump to the instructions:

 

Offline / Online Fabric Plane

The following steps are an example of offlining and onlining a plane in a MX960 router. They are applicable to MX480 and MX240.
  1. Identify the status of each plane with the following command, and ensure that you have spare planes available for redundancy:

    user@host> show chassis fabric summary.

    For example:

    user@host> show chassis fabric summary
    Plane State Uptime
    0 Online 3 hours, 7 minutes, 9 seconds
    1 Check  3 hours, 7 minutes, 4 seconds
    2 Fault  3 hours, 6 minutes, 59 seconds
    3 Online 3 hours, 6 minutes, 54 seconds

    4 Spare  3 hours, 7 minutes, 4 seconds
    5 Spare  3 hours, 7 minutes, 4 seconds

  2. Offline the plane with the following command:

    user@host> request chassis fabric plane offline <plane #>

    When the above command is executed, the corresponding plane will be offlined and barred from switching data traffic. The next numerically lowest spare-plane is made online and used for switching the packets.

    For example, in step 1, Plane 1 is in check state; it can be offlined as follows:

    user@host> request chassis fabric plane offline 1
    Offline initiated, use "show chassis fabric plane" to verify

  3. Verify if the plane is offlined, by running the 'show chassis fabric summary' command again:
    user@host> show chassis fabric summary
    Plane State Uptime
    0 Online 3 hours, 7 minutes, 9 seconds
    1 Offline
    2 Online 3 hours, 6 minutes, 59 seconds
    3 Online 3 hours, 6 minutes, 54 seconds
    4 Online 3 hours, 7 minutes, 4 seconds
    5 Spare  3 hours, 7 minutes, 4 seconds


    Notice that Plane 1 is offlined now, and that Plane 4 has moved from the Spare state to Online state and will be used to forward traffic in place of Plane 1.
  4. Online the plane with the following command:
    user@host> request chassis fabric plane online <plane #>
     

    When this command is executed, the corresponding plane will be onlined and placed in the Spare state. This plane will be spare, until the next time the numerically higher planes encounter any issue or when they are manually offlined. At that time, this plane will be made online and will start switching traffic.

    For example, in Step 3, plane 3 is in the Offline State. It is brought online and moved to spare as follows:

    user@host> request chassis fabric plane online 1
    Online initiated, use "show chassis fabric plane" to verify

  5. Verify if the plane is back online, by running the 'show chassis fabric summary' command:

    user@host> show chassis fabric summary 
    Plane  State Uptime
    0 Online 3 hours, 7 minutes, 9 seconds
    1 Spare  0 hours, 0 minutes, 4 seconds
    2 Online 3 hours, 6 minutes, 59 seconds
    3 Online 3 hours, 6 minutes, 54 seconds
    4 Online 3 hours, 7 minutes, 4 seconds
    5 Spare  3 hours, 7 minutes, 4 seconds

    From the above output, clearing the Check state on Plane 1 was successful.

    Optionally, Plane 1 can be put into the Online state again by offlining/onlining Plane 4.



 

Offline/Online Switch Control Board(SCB):

If offline/online of individual Fabric Planes does not resolve the issue, then the next step to offline/online the SCB is commonly done.

 

The following steps are an example of offlining and onlining a SCB in a MX960 router. They are applicable to MX480 and MX240.

On the MX960 router, two SCB's provide full fabric capacity for packet switching between PFEs (assuming only DPC cards) and the third SCB is spare. On the MX480 and MX240 routers, one SCB provides full fabric capacity for packet switching and the second SCB is spare.


Note: For MX960 routers, if the SCB in question is SCB2 (spare SCB which resides in slot8 and it does not have an RE), the SCB can be offlined/onlined directly without worrying about RE primary role.

 

  1. Identify the status of each plane with the following command, and ensure that you have spare planes available for redundancy:

    user@host> show chassis fabric summary.


    For example:
    > user@host> show chassis fabric summary
    Plane State Uptime
    0 Online 3 hours, 7 minutes, 9 seconds
    1 Check  3 hours, 7 minutes, 4 seconds
    2 Online 3 hours, 6 minutes, 59 seconds
    3 Online 3 hours, 6 minutes, 54 seconds

    4 Spare  3 hours, 7 minutes, 4 seconds
    5 Spare  3 hours, 7 minutes, 4 seconds
  2. Then identify the plane location of the fabric plane, which is in check or fault state, by using the command: show chassis fabric plane-location. This command indicates which SCB to offline.

    For example:

    user@host> show chassis fabric plane-location
    ------------Fabric Plane Locations-------------
    Plane 0                     Control Board 0
    Plane 1                     Control Board 0
    Plane 2                     Control Board 1
    Plane 3                     Control Board 1
    Plane 4                     Control Board 2
    Plane 5                     Control Board 2

    In the above two example outputs, you can see that Plane1 is in check state and residing in Control Board 0.

  3. If the SCB to be offlined is one which holds the Primary RE, then switch RE primary role. In order to determine if the SCB holds the primary RE, first identify the routing-engine primary role on the router by using the command: show chassis routing-engine.
    user@host> show chassis routing-engine 
    Routing Engine status:
      Slot 0:
        Current state                  Master
        Election priority              Master (default)
        Temperature                 40 degrees C / 104 degrees F
        CPU temperature             44 degrees C / 111 degrees F
        DRAM                      3584 MB
        Memory utilization          10 percent
        CPU utilization:
          User                       0 percent
          Background                 0 percent
          Kernel                     3 percent
          Interrupt                  1 percent
          Idle                      96 percent
        Model                          RE-S-2000
        Serial ID                      9012001461
        Start time                     2012-03-31 18:47:07 PST
        Uptime                         46 minutes, 37 seconds
        Last reboot reason             could not be determined
        Load averages:                 1 minute   5 minute  15 minute
                                           0.03       0.02       0.00
    Routing Engine status:
      Slot 1:
        Current state                  Backup
        Election priority              Backup (default)
        Temperature                 37 degrees C / 98 degrees F
        CPU temperature             42 degrees C / 107 degrees F
        DRAM                      3584 MB
        Memory utilization          13 percent
        CPU utilization:
          User                       0 percent
          Background                 0 percent
          Kernel                     0 percent
          Interrupt                  0 percent
          Idle                      99 percent
        Model                          RE-S-2000
        Serial ID                      9009037021
        Start time                     2012-03-31 19:17:16 PST
        Uptime                         16 minutes, 30 seconds
        Last reboot reason             0x11:power cycle/failure misc hardware reason


    In the above output, RE0 is primary and RE1 is backup.
    Control Board 0, the location for SCB to be offlined, corresponds to Slot 0.
     

    If the SCB to be offlined is holding the routing-engine which is the primary, then perform a primary role switch with the command: request chassis routing-engine master switch. In the above output example, SCB0 is to be offlined and is holding RE0, which is the primary. So you can perform a RE primary role switch:

    user@host> request chassis routing-engine master switch
    warning: Traffic will be interrupted while the PFE is re-initialized
    Toggle mastership between routing engines ? [yes,no] (no) yes

    Confirm that the Primary Role is switched by using the show chassis routing-engine command.
  4. Power-off the backup routing-engine with the command: request system power-off other-routing-engine

    For example, while logged into RE1 which is primary, the backup RE0 is powered off:

    user@host> request system power-off other-routing-engine

    Verify that the routing-engine is powered-off with the command:   show chassis routing-engine:
    user@host> show chassis routing-engine 
    Routing Engine status:
    Slot 0:     
     Current state                  Present           
    Slot 1:     
     Current state                  Master
        Election priority              Backup (default)
        Temperature                 37 degrees C / 98 degrees F
        CPU temperature             42 degrees C / 107 degrees F
        DRAM                      3584 MB
        Memory utilization          13 percent
        CPU utilization:
          User                       0 percent
          Background                 0 percent
          Kernel                     0 percent
          Interrupt                  0 percent
          Idle                      99 percent
        Model                          RE-S-2000
        Serial ID                      9009037021
        Start time                     2012-03-31 19:17:16 PST
        Uptime                         16 minutes, 30 seconds
        Last reboot reason             0x11:power cycle/failure misc hardware reason
    
  5. Offline the SCB with the command: request chassis cb slot offline

    In this example, CB0 is offlined:

    user@host> request chassis cb offline slot 0
    Offline initiated, use "show chassis environment cb" to verify
     Verify that CB0 is offlined:
    user@host> show chassis environment cb
    CB 0 status:
    State Offline

    Power 1 Disabled
    Power 1 Disabled
    CB 1 status:
    State Online Master

    Temperature 31 degrees C / 87 degrees F
    Power 1
    1.2 V 1205 mV
    1.25 V 1256 mV
    1.5 V 1511 mV
    1.8 V 1798 mV
    2.5 V 2507 mV
    3.3 V 3312 mV
    3.3 V SM3 3293 mV
    5.0 V 5033 mV
    5.0 V RE 5085 mV
    12.0 V 12084 mV
    12.0 V RE 11930 mV
    Power 2
    4.6 V bias MidPlane 4821 mV
    11.3 V bias PEM 11330 mV
    11.3 V bias FPD 11330 mV
    11.3 V bias POE 0 11292 mV
    11.3 V bias POE 1 11330 mV
    Bus Revision 42
    FPGA Revision 1

    The planes on the SCB are offlined and the planes in spare are switched to online. Verify the same with the command:  show chassis fabric summary

    user@host> show chassis fabric summary
    Plane State Uptime
    0 Offline
    1 Offline

    2 Online 3 hours, 6 minutes, 59 seconds
    3 Online 3 hours, 6 minutes, 54 seconds
    4 Online 3 hours, 7 minutes, 4 seconds
    5 Online 3 hours, 7 minutes, 4 seconds
  6. Wait for a few seconds, and then online the SCB, by using the command: show chassis online slot
    user@host> request chassis cb online slot 0
    Online initiated, use "show chassis environment cb" to verify
     

    The planes on this SCB are onlined and placed in Spare. Verify the same by using the show chassis fabric summary command:

    user@host> show chassis fabric summary
    Plane State Uptime
    0 Spare 3 hours, 7 minutes, 9 seconds
    1 Spare 3 hours, 7 minutes, 4 seconds
    2 Online 3 hours, 6 minutes, 59 seconds
    3 Online 3 hours, 6 minutes, 54 seconds
    4 Online 3 hours, 7 minutes, 4 seconds
    5 Online 3 hours, 7 minutes, 4 seconds
    Plane 0 and 1 will in Spare State, until one of the Online Planes are offlined or encounter any issue.
  7.  Power on the Routing-Engine, which was powered-off in Step 4 with the command: request system power-on other-routing-engine

    While logged into RE1 which is primary, the other RE is powered on:

    user@host> request system power-on other-routing-engine
    Verify that the Backup Routing is online by using the command: show chassis routing-engine.

    If you want to make the current Back-up RE (in our example RE0) as Primary again, use the command request chassis routing-engine master acquire on the back-up.



 
Modification History:
2021-03-19: Updated the article terminology to align with Juniper's Inclusion & Diversity initiatives.
Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search