Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[SRX/IDP] How to detect high CPU usage, collect data, and disable IDP from HE SRX for high CPU issue due to IDP

0

0

Article ID: KB26927 KB Last Updated: 23 Jul 2018Version: 3.0
Summary:

Data gathering for intermittent and service affecting issues can be a difficult task. The article provides a script for detecting high CPU usage, collect data, then disable the IDP device from the HE SRX device when high CPU usage is due to the IDP service.

Symptoms:

The script detects high CPU usage when the CPU threshold crosses 90%. When it detects over 90% of CPU utilization, it collects data every minute for 4 minutes, then disables the IDP device.

The data that the script collects depends on what is included in the srx3k-spu-cli/srx5k-spu-cli file. If more data has to be collected, the commands can be included in these files as required.

Solution:

Procedure:​

  1. Download the three attachments (catch.sh, srx3k-spu-cli and cli_config.xml) available at the foot note of this KB.

  2. Name or rename the file as srx3k-spu-cli if planing to use it for SRX 3000 series. Name or rename the file as srx5k-spu-cli if planning to use it for SRX 5000 series.

    Note: The file should not contain any extensions such as .txt or .rtf or .doc as the script would not read such files.

  3. Copy and paste the commands listed to the srx3k-spu-cli or srx5k-spu-cli file present on your desktop.

  4. Update the PICs list in the catch.sh script with the PICs available on your device. This can be done by running the tnpdump command on the shell to list the PICs we would be working with.

    Example:

    root@fw% tnpdump
    
    Name               TNPaddr   MAC address       IF     MTU E H R
    cluster7.node0     0x1700001 02:00:00:01:00:04 em0    1500 0 0 3
    cluster7.node0     0x1700001 02:00:01:01:00:04 em1    1500 0 1 3
    node0.re0          0x1700004 02:00:00:01:00:04 em0    1500 0 0 3
    node0.re0          0x1700004 02:00:01:01:00:04 em1    1500 0 1 3
    node0.fpc0         0x1700010 02:00:00:01:00:10 em0    1500 4 0 3
    node0.fpc4         0x1700014 02:00:00:01:00:14 em0    1500 5 0 3
    node0.fpc4.pic0    0x1700114 02:00:00:01:01:14 em0    1500 3 0 3 <<<
    node0.fpc4.pic1    0x1700214 02:00:00:01:02:14 em0    1500 2 0 3 <<<
     
    In catch.sh File:
    pics="
    node0.fpc4.pic0
    node0.fpc4.pic1
    "

    Note: The script catch.sh would execute the in-build SRX script (srx-cprod.sh) with the commands listed in srx3k-spu-cli or srx5k-spu-cli. Hence, the listed SPU PICs must be correct.

  5. The cli_config.xml file would be executed to disable the IDP configuration.

  6. Transfer the 3 files (catch.sh, srx3k-spu-cli/srx5k-spu-cli and cli_config.xml) to the /var/tmp/ folder. Do not change the name or extension of the files.

    Make sure to use the text mode, when transferring the files via WinSCP; otherwise certain characters will get appended in the script and files.
    Note: After the files are transferred, perform a vi on the files and make sure that no extra characters are appended to the lines present inside the file. 

    Alternately, you can use vi on SRX, then manually copy and paste the contents of the file to a newly created file on the box. 

    Example:

    > start shell user root
    % cd /var/tmp/​
    % vi srx3k-spu-cli 

    (Press 'i' to enter the edit/insert mode)
    <Paste the commands>
    (Press 'Esc' and enter ":wq" without the double quotes to write/save and quit)

    % vi catch.sh
    (Press 'i' to enter the edit/insert mode)
    <Paste all the line present in the updated script>
    (Press 'Esc' and enter ":wq" without the double quotes to write/save and quit)

    % vi cli_config.xml
    (Press 'i' to enter the edit/insert mode)
    <Paste all the line present in the xml file>
    (Press 'Esc' and enter ":wq" without the double quotes to write/save and quit)

    Once created and edit view the file using vi <filename> again make sure that no extra characters are appended to the lines present inside the file. 

    Example:

    % vi srx3k-spu-cli   (Followed by 'Esc' and then ":q" to quit)
    % vi catch.sh  (Followed by 'Esc' and then ":q" to quit)
    % vi cli_config.xml (Followed by 'Esc' and then ":q" to quit)
  7. Now go to /var/tmp/ in the shell and run the nohup sh catch.sh > & testfile.log & command to run the script in the background. Use the kill -9 <process-id> command to kill the script. To run the script in the foreground, run the sh catch.sh command.

    >start shell user root
    % cd /var/tmp/
    % nohup sh catch.sh > & testfile.log &

    [1] 57672
    % kill -9 57672


    (or)

    >start shell user root
    % cd /var/tmp/
    % sh catch.sh
  8. The logs will be saved in a file with the name that contains year, month, date, hour, minute and second.


The commands that need to be copied to the srx3k-spu-cli or srx5k-spu-cli file are as follows:
show version
show xlr pkt_mbuf
show xlr pkt_mbuf timeout
show xlr pkt_mbuf use
show mbuf host
show mbuf counter
show mbuf timeout
show arena
show services objcache
show usp idp status
show usp idp const subs s0
show usp idp const
show usp idp debug-counter memory
show usp idp debug-counter flow
show usp idp debug-counter ids
show usp idp debug-counter ai
show usp idp debug-counter kpp
show usp idp debug-counter reass
show usp idp debug-counter action
show usp idp attack-table
show usp idp context hit
show usp idp dfa-stats
show usp appid status
show usp appid config
show xlr cpu
test watchdog snapshot
show watchdog snapshot
show xlr cpu detail all
show usp flow stats
show usp flow counters all
show usp flow session summary
show xlr session detail

For example:
root@srx-3400% sh catch.sh
======= Thu Feb 14 17:13:58 UTC 2013========
<<<<<------- fpc6.pic0 --------->>>>>
Collect Data Now!
DATA GATHERING
DATA GATHERING
DATA GATHERING
DATA GATHERING
Disable IDP Now!
<routing-engine junos:style="normal">
<name>re0</name>
<commit-success/>

After running the script, notice that the IDP device will be disabled:

root@srx-3400> show configuration | display set | match deactivate
deactivate security idp

As you can see above, the script runs the deactivate security idp command to disable the IDP device.

Notes:

  • The process ID must be noted after, running the script with the 'nohup sh catch.sh > & testfile.log &' command; else the scripts will continuously run in the background, if the process ID is unknown or lost. Instead, customers can instead run the 'sh catch.sh' command to avoid having to note down the process ID.

  • The script is applicable for only SRX3000 and SRX5000 hardware models.

The script files are as follows:

Modification History:

2018-07-23: Added more details to the steps in the solution, more examples, and a file creation example to overcome the character appending issue.

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search