Data Collection and Troubleshooting Guides can help with issue investigation as well as reduce time to resolve. Each problem/issue could require a different set of data to collect. This article contains a list of data to collect as well as pointers to Resolution Guides and references on how to collect the data.
This section contains the following:
Caveats and tips:
- Traceoptions require additional system resources to gather and store data:
- Please ensure you have enough disk space when enabling traceoptions
- Gauge current system utilization before enabling traceoptions
- “
show chassis routing-engine”
- “
show security monitoring fpc <spc-slot>” (use slot 0 for branch)
- Instead of using “flag all” you can flag specific areas of interest
- Delete all traceoptions that are not needed for immediate debugging
- Don’t forget to remove traceoptions after data collection is completed. This can be done by deactivating or deleting the traceoptions configuration stanza that you previously added to activate traceoptions.
For example, let's say that traceoptions that you have enabled was configured as follows:
set chassis cluster traceoptions file cluster.tr size 5m files 5 world-readable
set chassis cluster traceoptions flag all
To disable this traceoptions configuration, you can issue one of the following two commands (not both) and commit the changes:
deactivate chassis cluster traceoptions
OR
delete chassis cluster traceoptions
- To deactivate paging (output stopping each page requiring you to press space bar) you can:
- Run “set cli screen-length 0” for this to apply for all commands for your sessions
- Add the “| no-more” option at the end of a command
Data to Collect for all configurations:
Regardless of configuration, all cases will benefit by attaching the session captures, request information output, and logs when initially opening the case. If you need to investigate an intermittent concern (for example slow transfers at peak hours) please be sure to collect this data at the time of the problem.
| All Configurations |
Background information |
- Provide all SSH / Telnet session captures
- Provide any available topology information
- Summary of how the device is being used (production, lab system, co-location, etc)
- Summary of device history (new install, production for X months/years, other recent cases, etc)
- Summary of any recent changes in the network or on the device
|
| Request support info |
- Enter:
request support information | save /var/log/rsi1.log
- Once step #1 completes, wait enough time to ensure that the condition you wish to address continues/appears before proceeding to the next step.
- Enter:
request support information | save /var/log/rsi2.log
|
| Logs |
Archive the /var/log/ contents:
file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz
To ensure the /var/log/ directory was properly archived, check the file size using the command: file list /var/tmp/CURRENT-DATE.tgz detail |
See
References section for the following:
Additional Data to Collect:
In addition, collect the data shown below for the following issues:
Jump to:
Chassis Cluster
Traffic failing for a specific host / application
High CPU
OSPF
BGP
Multicast
ALG
UTM - Anti-Virus
UTM - Anti-Spam
UTM - Web Filtering
UTM - Content Filtering
IPSec - Route Based
IPSec - Policy Based
IPSec - Dynamic VPNM
IPSec- NCP Exclusive Remove Access Client Connections
IDP - Security Package Update
IDP - Policy update after commit
IDP - Attack Detection
| Chassis Cluster |
Show commands: |
set cli timestamp
show chassis fpc pic-status
show chassis cluster status
show chassis cluster interfaces
show chassis cluster statistics
show chassis cluster information
show chassis cluster ip-monitoring status |
| Logs -Each node: |
/var/log/messages
/var/log/jsrpd
/var/log/chassisd |
Traceoptions:
Caveats |
set chassis cluster traceoptions file cluster.tr size 5m files 5 world-readable
set chassis cluster traceoptions flag all |
| Known Issues: |
List PRs |
[
Back to Top]
| Traffic failing for a specific host / application: |
Show commands: |
show security flow session summary
show security flow session {source-prefix | destination-prefix | source-port |
destination-port} <ip-prefix> extensive
show security flow session session-identifier <session-id> (same output as above)
show security flow cp-session summary
show interface extensive
show arp no-resolve (for locally connected hosts)
traceroute <ip-prefix> (for failing host) |
| Logs: |
· None by default.
· If security policy logs are enabled, check the configured log file for policy RT_FLOW events |
Traceoptions:
Caveats |
set security flow traceoptions file flow.trace
set security flow traceoptions file size 5m
set security flow traceoptions file files 5
set security flow traceoptions flag basic-datapath
set security flow traceoptions flag packet-drops
set security flow traceoptions packet-filter hostinit source-prefix a.a.a.a/32
set security flow traceoptions packet-filter hostinit destination-prefix b.b.b.b/32
set security flow traceoptions packet-filter hostresp source-prefix b.b.b.b/32
set security flow traceoptions packet-filter hostresp destination-prefix c.c.c.c/32
a.a.a.a - source address for initial traffic (use "inside/local" address if using source NAT)
b.b.b.b - destination address for initial traffic and source address for return traffic
c.c.c.c - destination address for return traffic (use "outside/global" address if using SRC NAT)
|
| Notes: |
See the ALG sections for more application-specific details. |
[
Back to Top]
| High CPU |
CLI commands: |
set cli timestamp
show chassis routing-engine
show system processes extensive
show system users
show system connections
show system statistics
show chassis forwarding
show security monitor fpc pic <SPC-slot> (use 0 for Branch platforms)
show security monitor performance spu
show security monitor performance sess |
| Logs: |
None |
| Traceoptions: |
None |
[
Back to Top]
| OSPF |
| Show commands: |
(If OSPF is running in a routing instance, specify which instance where applicable)
set cli timestamp
show ospf overview
show ospf database
show ospf neighbor detail
show ospf route
show ospf statistics
show ospf interface
show ospf log
show route protocol ospf
show route <x.x.x.x> extensive
show ospf database extensive
|
| Logs: |
/var/log/messages |
Traceoptions:
Caveats |
(use below for inet.0 default instance)
set protocols ospf traceoptions file ospf.tr
set protocols ospf traceoptions file size 5m
set protocols ospf traceoptions file files 5
set protocols ospf traceoptions flag all
(use below for routing instances)
set routing-instances ospf-vr protocols ospf traceoptions file ospf-vr.tr
set routing-instances ospf-vr protocols ospf traceoptions file size 5m
set routing-instances ospf-vr protocols ospf traceoptions file files 5
set routing-instances ospf-vr protocols ospf traceoptions flag all |
| Known Issues: |
List PRs |
[
Back to Top]
| BGP |
Show commands: |
set cli timestamp
show bgp summary
show bgp neighbor
show route advertising-protocol bgp <neighbor-address> extensive
show route receive-protocol bgp <neighbor-address>
show route forwarding-table
show route resolution unresolved |
| Logs: |
/var/log/messages |
Traceoptions:
Caveats |
set protocols bgp traceoptions file bgp.tr
set protocols bgp traceoptions file size 5m
set protocols bgp traceoptions file files 5
set protocols bgp traceoptions flag all
set routing-instances bgp-vr protocols bgp traceoptions file bgp.tr
set routing-instances bgp-vr protocols bgp traceoptions file size 5m
set routing-instances bgp-vr protocols bgp traceoptions file files 5
set routing-instances bgp-vr protocols bgp traceoptions flag all |
| Known Issues: |
List PRs |
[
Back to Top]
| Multicast |
Show commands: |
show multicast route
show multicast statistics
show multicast sessions
show multicast usage
show multicast interface
show multicast next-hops
show multicast rpf summary
show interface <if-name> extensive
show igmp group detail
show igmp statistics
show igmp interface detail
show pim statistics
show pim neighbors
show pim rps detail
show pim join extensive
show pim bootstrap
show msdp source-active
show msdp detail
show msdp statistics
show route |
| Logs: |
/var/log/messages |
Traceoptions:
Caveats |
set routing-options multicast traceoptions file mcast.tr
set routing-options multicast traceoptions file size 5m
set routing-options multicast traceoptions file files 5
set routing-options multicast traceoptions flag all |
| Known Issues: |
List PRs |
[
Back to Top]
| ALG |
CLI commands: |
set cli timestamp
show security alg status
show security alg <alg-name>
[obtain all sub-commands for the ALG in question,
use “?” to view options]
show security resource-manager summary
show security resource-manager resource active
show security resource-manager resource active <number>
show security resource-manager group active
show security resource-manager group active <number>
show security flow gate |
| Logs: |
none |
Traceoptions:
Caveats |
set security traceoptions file alg-sec.tr
set security traceoptions file size 5m
set security traceoptions file files 5
set security traceoptions file world-readable
set security traceoptions flag all
set security alg <alg-type> traceoptions flag all
set security flow traceoptions file alg-flow.tr
set security flow traceoptions file size 5m
set security flow traceoptions file files 5
set security flow traceoptions file world-readable
set security flow traceoptions flag basic-datapath
set security flow traceoptions flag packet-drops
set security flow traceoptions packet-filter alginit source-prefix a.a.a.a/32
set security flow traceoptions packet-filter alginit destination-prefix b.b.b.b/32
set security flow traceoptions packet-filter algresp source-prefix b.b.b.b/32
set security flow traceoptions packet-filter algresp destination-prefix c.c.c.c/32 |
| Notes: |
See "Traffic failing for a specific host/application" for an example of packet filters for flow traceoptions. |
| Known Issues: |
List PRs |
[
Back to Top]
| UTM - Anti-Virus |
CLI commands: |
set cli timestamp
show system licenses
show security utm status
show security utm session
show security utm anti-virus status detail
show security utm anti-virus statistics
show chassis routing-engine
show system processes extensive
Updating Full AV database:
request security utm anti-virus kaspersky-lab-engine pattern-update
Updating Express AV database:
request security utm anti-virus juniper-express-engine pattern-update
Updating Sophos AV database:
request security utm anti-virus sophos-engine pattern update |
| Logs: |
/var/log/utmd
/var/log/utmd-av |
Traceoptions:
Caveats |
set security utm traceoptions flag all
set security utm application-proxy traceoptions flag all
set security utm feature-profile anti-virus traceoptions flag all
set security traceoptions file av.tr
set security traceoptions file size 5m
set security traceoptions file files 5
set security traceoptions file world-readable
set security traceoptions flag all |
| Known Issues: |
List PRs |
[
Back to Top]
| UTM - Anti-Spam |
CLI commands: |
set cli timestamp
show system licenses
show security utm status
show security utm session
show security utm anti-spam status
show security utm anti-spam statistics
show chassis routing-engine
show system processes extensive |
| Logs: |
/var/log/utmd
/var/log/utmd-as |
Traceoptions:
Caveats |
set security utm traceoptions flag all
set security utm application-proxy traceoptions flag all
set security utm feature-profile anti-spam traceoptions flag all
set security traceoptions file as.tr
set security traceoptions file size 5m
set security traceoptions file files 5
set security traceoptions file world-readable
set security traceoptions flag all |
| Known Issues: |
List PRs |
[
Back to Top]
| UTM - Web Filtering |
CLI commands: |
set cli timestamp
show system licenses
show security utm status
show security utm session
show security utm web-filtering status
show security utm web-filtering statistics
show chassis routing-engine
show system processes extensive |
| Logs: |
/var/log/utmd
/var/log/utmd-wf |
Traceoptions:
Caveats |
set security utm traceoptions flag all
set security utm application-proxy traceoptions flag all
set security utm feature-profile web-filtering traceoptions flag all
set security traceoptions file wf.tr
set security traceoptions file size 5m
set security traceoptions file files 5
set security traceoptions file world-readable
set security traceoptions flag all |
| Known Issues: |
List PRs |
[
Back to Top]
| UTM - Content Filtering |
CLI commands: |
set cli timestamp
show system licenses
show security utm status
show security utm session
show security utm content-filtering statistics |
| Logs: |
/var/log/utmd |
Traceoptions:
Caveats |
set security utm traceoptions flag all
set security utm application-proxy traceoptions flag all
set security utm feature-profile content-filtering traceoptions flag all
set security traceoptions file cf.tr
set security traceoptions file size 5m
set security traceoptions file files 5
set security traceoptions file world-readable
set security traceoptions flag all |
| Known Issues: |
List PRs |
[
Back to Top]
IPsec VPN -
Route-Based |
Show commands: |
show security ike security-association
show security ike security-association index <#> detail
show security ipsec security-association
show security ipsec security-association index <#> detail
show security ipsec statistics
show security ipsec statistics index <#>
show security ipsec next-hop-tunnels
monitor interface st0.x
show interfaces extensive st0.x
show security flow session tunnel
show route
show security pki local-cert detail
show security pki ca-cert detail
show security pki crl detail |
| Logs: |
/var/log/kmd*
/var/tmp/kmd* (SRX 1400 and higher) |
Traceoptions:
Caveats |
set security ike traceoptions file vpn.tr size 5m files 5 world-readable
set security ike traceoptions flag ike
set security ike traceoptions flag general
set security ipsec traceoptions flag security-associations
set security ipsec traceoptions flag packet-drops
set security ipsec traceoptions flag packet-processing |
| Notes: |
If tunnels are up but traffic not passing, see section “Traffic failing for a specific host/application” and
setup packet filters for outer ESP traffic as well as inner application/host traffic. |
[
Back to Top]
| IPsec VPN - Policy-Based |
Show commands: |
show security ike security-association
show security ike security-association index <#> detail
show security ipsec security-association
show security ipsec security-association index <#> detail
show security ipsec statistics
show security ipsec statistics index <#>
show security ipsec next-hop-tunnels
show security flow session tunnel
IF PKI certs are used:
show security pki local-cert detail
show security pki ca-cert detail
show security pki crl detail
show security policies detail
show log /var/etc/policy.id |
| Logs: |
/var/log/kmd*
/var/tmp/kmd* (SRX 1400 and higher) |
Traceoptions:
Caveats |
set security ike traceoptions file vpn.tr size 5m files 5 world-readable
set security ike traceoptions flag ike
set security ike traceoptions flag general
set security ipsec traceoptions flag security-associations
set security ipsec traceoptions flag packet-drops
set security ipsec traceoptions flag packet-processing |
| Notes: |
If tunnels are up but traffic not passing, see section “Traffic failing for a specific host/application” and setup packet filters for outer ESP traffic as well as inner application/host traffic. |
[
Back to Top]
| IPsec - Dynamic VPN |
Show commands: |
show security ike security-association
show security ike security-association index <number> detail
show security ike active-peer
show security ipsec security-association
show security ipsec security-association index <id>
show security ipsec statistics
show security dynamic-vpn client version
show security dynamic-vpn users detail
show system license |
Logs: |
SRX:
/var/log/kmd
/var/log/httpd.log
/var/log/authd
Pulse client:
File > Logs > Log level > (detailed / normal)
File > Logs > Save as > <filename> |
Traceoptions:
Caveats |
set system processes general-authentication-servic traceoptions file dynvpn-auth.tr
set system processes general-authentication-servic traceoptions file size 5m
set system processes general-authentication-servic traceoptions file files 5
set system processes general-authentication-servic traceoptions file world-readable
set system processes general-authentication-servic traceoptions flag all
set security ike traceoptions file dynvpn.tr size 5m files 5 world-readable
set security ike traceoptions flag ike
set security ike traceoptions flag general
set security ipsec traceoptions flag security-associations
set security ipsec traceoptions flag packet-drops
set security ipsec traceoptions flag packet-processing |
| Notes: |
If tunnels are up but traffic not passing, see section “Traffic failing for a specific host/application” and setup packet filters for outer ESP traffic as well as inner application/host traffic.
The Pulse client version is also helpful for troubleshooting. For more info on how to get it, see KB22857 - How to: Find the Junos Pulse version in the SRX and in the Pulse client itself. |
| Known Issues: |
List PRs |
[
Back to Top]
IPSec VPN -
NCP Exclusive Remote Access Client Connections |
Show commands: |
show security ike active-peer
show security ike security-association
show security ike security-association index <#> detail
show security ipsec security-association
show security ipsec security-association index <#> detail
show security ipsec tunnel-events-statistics
show security ipsec statistics
show security ipsec statistics index <#>
show interfaces extensive st0.x
show security flow session tunnel
show route
show security pki local-cert detail
show security pki ca-cert detail
show security pki crl detail
show network-access requests statistics
show system license |
| Logs: |
/var/log/kmd* |
Traceoptions:
Caveats |
set security ike traceoptions file vpn.tr size 5m files 5 world-readable
set security ike traceoptions flag ike
set security ike traceoptions flag general
set security ipsec traceoptions flag security-associations
set security ipsec traceoptions flag packet-drops
set security ipsec traceoptions flag packet-processing |
| Notes: |
If tunnels are up but traffic not passing, see section “Traffic failing for a specific host/application” and
setup packet filters for outer ESP traffic as well as inner application/host traffic. |
[
Back to Top]
| IDP - Security Package Update |
Show commands: |
show security idp security-package-version
show security idp status
show security idp memory
request security idp security-package download
request security idp security-package download status
request security idp security-package install
request security idp security-package install status |
Logs: |
show log messages
show log idpd
show log idp-traceoptions |
Traceoptions:
Caveats |
edit security idp traceoptions
set file idp-traceoptions
set flag all
set level all
edit security flow traceoptions
set file flow-trace
set flag basic-datapath
set flag packet-drops
set packet-filter 1 …
ALWAYS CONFIGURE PACKET-FILTERS |
[
Back to Top
| IDP - Policy update after commit |
Show commands: |
show security idp security-package-version
show security idp status
show security idp policies
show security idp policy-commit-status
show security idp policy-templates-list
show security idp counters policy-manager
show security idp memory
show security idp policy-commit-status clear |
Logs: |
show log messages
show log idpd
show log idp-traceoptions
show log /tmp/idpd_trace |
Traceoptions:
Caveats |
edit security idp traceoptions
set file idp-traceoptions
set flag all
set level all
edit security flow traceoptions
set file flow-trace
set flag basic-datapath
set flag packet-drops
set packet-filter 1 … ALWAYS CONFIGURE PACKET-FILTERS |
| Notes: |
Successful IDP policy installation displays IDP_POLICY_LOAD_SUCCEEDED in both idpd and traceoptions log files. |
[
Back to Top]
| IDP - Attack Detection |
Show commands: |
show security idp security-package-version
show security idp status
show security idp counters flow
show security idp counters application-identification
show security idp counters flow
show security idp counters log
show security idp counters packet
show security idp memory
show security idp application-statistics
show security idp attack table |
| Notes: |
Latency/Performance:
Change IDP policy to one of the predefined template IDP policies, like Recommended Policy, to verify if this is a customer IDP policy issue.
Datasheet benchmarks are based on IDP Recommended Policy.
False Positives/Negatives:
Gather:
1) Packet capture of the False Positive/Negative
2) IDP signature that is causing issue
3) show security idp security-package-version
4) show configuration security idp | display set
Contact signatures@juniper.net with above info |
[
Back to Top]
References:
How to:
Resolution Guides and Troubleshooting Checklists:
JTAC Certified step-by-step troubleshooting flowcharts and articles
Resolution
Guides and Articles - SRX - NAT
Troubleshooting Checklist - SRX - UTM
Troubleshooting Checklist - SRX - DHCP
Troubleshooting Checklist - SRX - RADIUS
Technical Bulletins: SRX Series (login to see more)
For more information on Technical Bulletins, see
KB9890 - How do I subscribe to a technical bulletin so I can I get email alerts regarding product issues, new product release announcements and security or safety issues?.