Data Collection and Troubleshooting Guides can help with issue investigation as well as reduce resolution time. Each issue might require a different set of data to collect. This article contains a list of data to collect, as well as pointers to Resolution Guides and references on how to collect the data. The goal is to reduce the time spent on initial data collection and reduce resolution time by providing a comprehensive list of what to collect or gather to troubleshoot an issue.
This section contains the following:
Caveats and tips:
Data to Collect for all configurations:
Regardless of configuration, all cases will benefit by attaching the session captures, request information output, and logs when initially opening the case. If you need to investigate an intermittent concern (for example slow transfers at peak hours) please be sure to collect this data at the time of the problem.
| All Configurations |
Background Information
|
- Provide all SSH / Telnet session captures.
- Provide any available topology information.
- Summary of how the device is being used (production, lab system, co-location, etc).
- Summary of device history (new install, production for X months/years, other recent cases, etc).
- Summary of any recent changes in the network or on the device.
|
|
Request Support Information
|
-
Enter: request support information | save /var/log/rsi1.log
-
After step #1 completes, wait enough time to ensure that the condition you wish to address continues/appears before proceeding to the next step.
-
Enter: request support information | save /var/log/rsi2.log
|
|
Logs
|
Archive the /var/log/ contents:
file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz
To ensure the /var/log/ directory was properly archived, check the file size using the command: file list /var/tmp/CURRENT-DATE.tgz detail
|
See
References section for the following:
Additional Data to Collect:
In addition, collect the data shown below for the following issues:
Jump to:
BGP
OSPF
RIP
ISIS
Multicast
MPLS
VPLS
L3 VPN
L2 VPN
CoS
Firewall Filters
IPv6
SNMP
High CPU
Fabric Plane (SBC, FChip, Plane)
NG-MPVN
CGNAT
Spanning Tree
BFD
| BGP |
Show Commands
|
set cli timestamp
show bgp summary
show bgp group
show bgp neighbor
show route next-hop database
show route protocol bgp
show route protocol bgp terse
show route receive-protocol bgp
show route advertising-protocol bgp
show route advertising-protocol bgp <neighbor-address> extensive
show route receive-protocol bgp <neighbor-address>
show route forwarding-table
show route resolution unresolved
show route hidden
show route community <reg-ex>
show route terse community-name <name>
show route damping
show route aspath-regex "regex"
show route damping [history|decayed|surpressed]
monitor traffic interface XX-X/X/X size 4096 detail
|
|
Logs
|
Archive /var/log as specified in Step 1 above |
|
Traceoptions
Caveats
|
set protocols bgp traceoptions file bgp.tr
set protocols bgp traceoptions file size 5m
set protocols bgp traceoptions file files 5
set protocols bgp traceoptions flag all
set routing-instances bgp-ri protocols bgp traceoptions file bgp.tr
set routing-instances bgp-ri protocols bgp traceoptions file size 5m
set routing-instances bgp-ri protocols bgp traceoptions file files 5
set routing-instances bgp-ri protocols bgp traceoptions flag all
|
[Back to Top]
| OSPF |
|
Show Commands
|
If OSPF is running in a routing instance, specify which instance, where applicable.
set cli timestamp
show ospf overview
show ospf database
show ospf neighbor detail
show ospf route
show ospf statistics
show ospf interface
show ospf log
show route protocol ospf
show route <x.x.x.x> extensive
show ospf database summary
show ospf database extensive
show bfd session
|
|
Logs
|
Archive /var/log as specified in Step 1 above |
|
Traceoptions
Caveats
|
Use below for inet.0 default instance:
set protocols ospf traceoptions file ospf.tr
set protocols ospf traceoptions file size 5m
set protocols ospf traceoptions file files 5
set protocols ospf traceoptions flag all
Use below for routing instances:
set routing-instances ospf-vr protocols ospf traceoptions file ospf-vr.tr
set routing-instances ospf-vr protocols ospf traceoptions file size 5m
set routing-instances ospf-vr protocols ospf traceoptions file files 5
set routing-instances ospf-vr protocols ospf traceoptions flag all
|
|
Known Issues
|
N/A
|
[Back to Top]
| RIP |
Show Commands
|
set cli timestamp
show rip overview
show rip neighbor
show route protocol rip
show route advertising-protocol rip <neighbor-address> extensive
show route receive-protocol rip <neighbor-address>
show rip statistics
show route forwarding-table
show route resolution unresolved |
|
Logs
|
Archive /var/log as specified in Step 1 above |
|
Traceoptions
Caveats
|
set protocols rip traceoptions file rip.tr
set protocols rip traceoptions file size 5m
set protocols rip traceoptions file files 5
set protocols rip traceoptions flag all
set routing-instances rip-vr protocols rip traceoptions file rip.tr
set routing-instances rip-vr protocols rip traceoptions file size 5m
set routing-instances rip-vr protocols rip traceoptions file files 5
set routing-instances rip-vr protocols rip traceoptions flag all |
[Back to Top]
| ISIS |
Show Commands
|
set cli timestamp
show isis adjacency
show isis adjacency extensive
show isis interface
show isis interface details
show isis hostname
show isis spflog
show isis statistics
show isis route
show route protocol isis
show isis database
show isis database detail
show bdf session
show route forwarding-table |
|
Logs
|
Archive /var/log as specified in Step 1 above |
|
Traceoptions
Caveats
|
set protocols isis traceoptions file rip.tr
set protocols isis traceoptions file size 5m
set protocols isis traceoptions file files 5
set protocols isis traceoptions flag all
set routing-instances isis-vr protocols isis traceoptions file isis.tr
set routing-instances isis-vr protocols isis traceoptions file size 5m
set routing-instances isis-vr protocols isis traceoptions file files 5
set routing-instances isis-vr protocols isis traceoptions flag all |
[Back to Top]
| Multicast |
Show Commands
|
General
show multicast route
show multicast statistics
show multicast sessions
show multicast usage
show multicast interface
show multicast next-hops
show multicast rpf summary
show interface <if-name> extensive
IGMP
show igmp group detail
show igmp statistics
show igmp interface detail
PIM
show pim statistics
show pim neighbors
show pim rps detail
show pim join extensive
show pim bootstrap
show pim source detail
MSDP
show msdp source-active
show msdp detail
show msdp statistics
|
|
Logs
|
Archive /var/log as specified in Step 1 above |
|
Traceoptions
Caveats
|
set protocols pim|igmp|msdp traceoptions file multicast-proto.tr
set protocols pim|igmp|msdp traceoptions file size 5m
set protocols pim|igmp|msdp traceoptions file files 5
set protocols pim|igmp|msdp traceoptions flag all
Enable traceoptions for PIM, IGMP, or MSDP.
|
[Back to Top]
| MPLS |
Show Commands
|
show mpls interface
show mpls lsp
show mpls lsp extensive
show mpls lsp ingress
show mpls lsp transit
show ted database
RSVP
show rsvp interface
show rsvp neighbor
show rsvp session
show rsvp session ingress
show rsvp version
show route table mpls.0
show route table inet.3
LDP
show ldp neighbors
show ldp session
show ldp database [session peer]
|
|
Logs
|
Archive /var/log as specified in Step 1 above |
|
Traceoptions
Caveats
|
set protocols mpls|ldp|rsvp traceoptions file mpls.tr
set protocols mpls|ldp|rsvp traceoptions file size 5m
set protocols mpls|ldp|rsvp traceoptions file files 5
set protocols mpls|ldp|rsvp traceoptions flag all
set protocols mpls label-switched-path <lsp-name> traceoptions file mpls-lsp.tr
set protocols mpls label-switched-path <lsp-name> traceoptions file size 5m
set protocols mpls label-switched-path <lsp-name> traceoptions file files 5
set protocols mpls label-switched-path <lsp-name> traceoptions flag all
|
[Back to Top]
| VPLS |
Show Commands
|
show vpls connections extensive
show bgp summary
show route table <instance-name>.l2vpn.0 extensive
show route table bgp.l2vpn.0 extensive
show route table mpls.0 extensive
show route table inet.3 extensive
show vpls flood extensive
show vpls mac-table
show interfaces terse
show interfaces routing
show route forwarding table family mpls extensive
show route forwarding table family vpls extensive
show interface ge-x/y/z extensive | no-more
show l2-learning interface
show l2-learning mac-move-buffer extensive
show l2-learning l2alm-peers
show l2-learning debug-statistics events
show l2-learning debug-statistics ipc
show l2-learning debug-statistics mac-events
show l2-learning debug-statistics mac-messages
show l2-learning debug-statistics mac-processing
show l2-learning debug-statistics performance-counters
show l2-learning debug-statistics rtsock
show bridge statistics
show vpls statistics
show bridge mac-table instance <instance-name> extensive
show vpls mac-table instance <instance-name> extensive
Determine if traffic is getting the VPLS interface, apply the following Firewall filter in the VPLS interface as input and output:
set firewall family vpls filter jtac-test term t1 from destination-mac-address <>
set firewall family vpls filter jtac-test term t1 from source-mac-address <>
set firewall family vpls filter jtac-test term t1 then count counter_in
set firewall family vpls filter jtac-test term t1 then accept
set firewall family vpls filter jtac-test term t2 then accept
root@root> show firewall filter jtac-test
Filter: jtac-test
Counters:
Name Bytes Packets
counter_in 0 0
Test PE to PE connectivity over the VPLS PW:
user@PE1> ping vpls instance red destination-mac <*> source-ip <*>
For the source-ip, configure IRB under the VPLS instance within the same address range within CPEs. Remember to have loopback interface open for the source address.
If M/T Series
show route forwarding-table family vpls vpn <instance-name> extensive
This command is used for the M/T series. It is equivalent to the previous command.
|
|
Logs
|
Archive /var/log as specified in Step 1 above |
|
Traceoptions
Caveats
|
If allowed, turn on the following traceoptions for approximately five minutes while the traffic is on. Then, turn it back off and collect the log files. See Caveats.
set routing-instances <instance-name> protocols vpls traceoptions flag all
This command will collect all vpls-related traces.
set protocols l2-learning traceoptions file l2ald-trace size 100m;
set protocols l2-learning traceoptions file flag mac-learning;
set protocols l2-learning traceoptions file flag ipc;
set protocols l2-learning traceoptions file flag routing-socket;
|
[Back to Top]
| L3 VPN |
CLI Commands
|
show route table <vpn-a>
show route table <vpn-a> hidden
show route forwarding-table vpn <vpn-a>
ping x.x.x.x routing-instance <vpn-a>
traceroute x.x.x.x routing-instance <vpn-a>
telnet x.x.x.x routing-instance <vpn-a>
ping mpls l3vpn <vpn-a> prefix <prefix>
show route table bgp.l3vpn.0
show route advertising-protocol bgp x.x.x.x
show route receive-protocol bgp x.x.x.x
show arp
OSPF VPN commands
show ospf interface instance <vpn-a>
show ospf neighbor instance <vpn-a>
show ospf database instance <vpn-a>
|
|
Logs
|
Archive /var/log as specified in Step 1 above |
|
Traceoptions
|
None
|
[Back to Top]
| L2 VPN |
CLI Commands
|
Kompella
show l2vpn connections extensive
show route table <vpn-a>
show route table bgp.l2vpn.0
show route advertising-protocol bgp x.x.x.x
show route receive-protocol bgp x.x.x.x
show route table mpls.o
show route forwarding-table family mpls
show rsvp session extensive
Martini
show l2circuit connections extensive
show ldp neighbor detail
show ldp database detail
show route table mpls.o
|
|
Logs
|
Archive /var/log as specified in Step 1 above |
|
Traceoptions
|
Kompella
set routing-instances l2vpn protocols l2vpn traceoptions file l2vpn_trace
set routing-instances l2vpn protocols l2vpn traceoptions file size 50m
set routing-instances l2vpn protocols l2vpn traceoptions file files 5
set routing-instances l2vpn protocols l2vpn traceoptions flag all
Martini
set protocols l2circuit traceoptions file l2circuit_trace
set protocols l2circuit traceoptions file size 50m
set protocols l2circuit traceoptions file files 5
set protocols l2circuit traceoptions flag all
set protocols ldp traceoptions file ldp_trace
set protocols ldp traceoptions file size 50m
set protocols ldp traceoptions file files 5
set protocols ldp traceoptions flag all
|
[Back to Top]
| CoS |
CLI Commands
|
show interface xx-x/x/x detail
show interface queue xx-x/x/x both-ingress-egress
show interfaces XX-X/X/X extensive
show class-of-service interfaces XX-X/X/X
show class-of-service code-point-aliases
show class-of-service code-point-aliases dscp
show class-of-service code-point-aliases inet-prec
show class-of-service code-point-aliases exp
show class-of-service forwarding-class
show class-of-service classifier [name]
show class-of-service scheduler-map
show class-of-service rewrite-rule
show class-of-service drop-profile
show class-of-service forwarding-table
show class-of-service forwarding-table classifier mapping
show class-of-service forwarding-table scheduler-map
|
|
Logs
|
Archive /var/log as specified in Step 1 above |
|
Traceoptions
|
None
|
[Back to Top]
| Firewall Filters |
CLI Commands
|
show interfaces filters
show firewall
show firewall filter <filter-name>
show firewall filter <name> prefix-action <psa-name> from 1 to 8
show firewall log
show log <log-file-name>
show policer
show interface policer XX-X/X/X |
|
Logs
|
Archive /var/log as specified in Step 1 above
show log /var/tmp/<sample file>
show log sampled |
|
|
Traceoptions
|
None
|
[Back to Top]
| IPv6 |
CLI Commands
|
set show interface terse
show route table inet6
show ipv6 neighbor
show interface lo0 extensive | display xml |
|
Logs
|
Archive /var/log as specified in Step 1 above |
|
Traceoptions
|
None
|
[Back to Top]
| SNMP |
CLI Commands
|
set cli timestamp
show configuration snmp
show snmp statistics extensive # Multiple time with polling activity.
show snmp mib walk <oid>
show snmp mib walk ipfragfails
show interface lo0 extensive
show system processes extensive | no-more # Multiple time with polling activity.
show chassis routing-engine | no-more # Multiple time with polling activity.
show system commit | no-more
show system statistics # Multiple time with polling activity.
show system buffers # Multiple time with polling activity.
show system core-dumps
show route <ip of NMS server>
ping <ip of NMS server> source <snmp source ip> count 500 rapid
show pfe statistics traffic # Multiple time with polling activity.
show interface extensive | no-more
show system process extensive | no-more
show snmp statistics subagents | no-more
show snmp stats-response-statistics |
|
Logs
|
Archive /var/log as specified in Step 1 above |
|
Traceoptions
|
set snmp traceoptions file snmp files 10 size 50m
set snmp traceoptions flag all
|
[Back to Top]
| High CPU |
CLI Commands
|
set cli timestamp
show chassis routing-engine (multiple snapshots, atleast 5)
show system processes extensive (multiple snapshots atleast 5)
show system users
show system connections
show system statistics
Turn on task accounting and collect the task accounting detail output (three times with a gap of 30 seconds). Don't forget to turn it off after finished.
set task accounting on
show task accounting detail
set task accounting off
show task memory detail
show task memeory summary
show task io
show task history
show task statistics
show task job
show task jobs
show krt queue
show krt state
|
|
Logs
|
Archive /var/log as specified in Step 1 above |
|
Traceoptions
|
user@router# show routing-options
traceoptions {
file routing-trace size 10m files 20 world-readable;
flag task;
flag state;
flag timer;
}
|
[Back to Top]
MX
Fabric
Plane
(SBC,
FChip,
Plane)
|
Show Commands
|
General
show chassis environment cb
show chassis alarms
Fabric-related
show chassis hardware
show chassis fabric summary
show chassis fabric map
show chassis fabric fpc
show chassis fabric plane
show class-of-service fabric statistics (take multiple outputs to see counters incrementing)
show pfe statistics traffic (take multiple outputs to see counters incrementing)
FPC Commands (from CLI)
request pfe execute target <fpc_no> command "show hsl2 statistics detail"
request pfe execute target <fpc_no> command "show syslog messages"
|
|
Logs
|
Archive /var/log as specified in Step 1 above
show log messages
show log chassisd |
|
|
Hidden Commands
Caveats
|
Note: These commands are not officially supported, and are hidden; however, they are good for troubleshooting Fabric Plane issues.
Exercise caution when executing these commands, as incorrect syntax may lead to problems. 15.1R7 may reset the FPC, but 18.4R2 is stable.
show chassis fabric asic-configuration
show chassis fabric statistics detail <plane_no> totals
Take multiple outputs to see counters incrementing and from all the planes.
show chassis hsl channel asic-name CBXFY slot Z extensive
X is the CB slot.
Y is the Fabric Plane number.
Z is the FPC slot no.
CBXFY is in upper case.
For example:
show chassis hsl channel asic-name CB0F1 slot 3 extensive
Run this command for all the SCB's and for all the planes multiple times to see incrementing counters.
|
|
Questions
|
- Have the SCB's for which the alarms have generated had any previous history of any issues?
- What kind of troubleshooting has already been done to mitigate the problem. Please provide detailed steps.
- Were there any recent changes to the configuration?
- Were there any changes to the hardware setup, or was new hardware introduced?
- Were there any events that occurred just before the log message that may have triggered it? This may include events such as routing engine switchover etc.
- Was a software upgrade/downgrade done recently?
|
[Back to Top]
| NG-MPVN |
Show Commands
|
show route table<instance-name>.inet.0 | no-more
show route table<instance-name>.mvpn.0 | no-more
show route table<instance-name>.inet.0 extensive | no-more
show route table<instance-name>.mvpn.0 extensive | no-more
show pim joininstance <instance-name> extensive | no-more
show multicast route instance <instance-name> extensive | no-more
show interface extensive |no-more
show pim rps instance <instance-name> extensive | no-more
show mpls lsp p2mp ingress |no-more
show mpls lsp p2mp ingress extensive |no-more
show mpls lsp p2mp | no-more
show mpls lsp | no-more
show bgp summary |no-more
|
|
Logs
(each node)
|
Archive /var/log as specified in Step 1 above |
|
Traceoptions
|
PIM detailed Traceoptions
RSVP detailed Traceoptions
BGP detailed Traceoptions
MVPN detailed Traceoptions
RTSOCKMON
|
|
Known Issues
|
List Open PR issues [change href values]
|
[Back to Top]
CGNAT
|
Show Commands
|
show chassis pic fpc-slot x pic-slot x
show services service-sets cpu-usage
show services service-sets summary
show services service-sets statistics syslog
show services service-sets statistics packet-drops
show services service-sets memory-usage
show services service-sets memory-usage zone
show services stateful-firewall conversations
show services stateful-firewall conversations destination-prefix
show services stateful-firewall statistics extensive
show services stateful-firewall flow-analysis
show services stateful-firewall flows extensive
show services nat mappings detail
show services nat pool detail
show services nat mappings endpoint-independent
show services nat mappings address-pooling-paired
|
|
Logs
|
Archive /var/log as specified in Step 1 above |
|
Traceoptions
|
None
|
[Back to Top]
Spanning
Tree
|
Show Commands
|
show spanning-tree bridge detail
show spanning-tree bridge msti x detail
show spanning-tree interface detail
show spanning-tree mstp configuration detail
show spanning-tree statistics bridge
show spanning-tree statistics interface detail
show spanning-tree statistics message-queues
show spanning-tree statistics routing-instance detail
show spanning-tree stp-buffer see-all
|
|
Logs
|
Archive /var/log as specified in Step 1 above |
|
Traceoptions
|
set protocols rstp|mstp|vstp traceoptions file spanning-tree.trace
set protocols rstp|mstp|vstp traceoptions file size 20m
set protocols rstp|mstp|vstp traceoptions file files 5
set protocols rstp|mstp|vstp traceoptions flag all
|
[Back to Top]
BFD
|
Show Commands
|
( take 4 outputs at 30 seconds apart)
show ppm connections detail
show bfd session extensive | no-more
show ppm interfaces detail | no-more
show ppm adjacencies detail | no-more
show ppm transmissions detail | no-more
show pfe statistics traffic | no-more
|
|
Logs
|
Archive /var/log as specified in Step 1 above |
|
Traceoptions
|
set routing-options ppm traceoptions file ppm
set routing-options ppm traceoptions file size 30m
set routing-options ppm traceoptions flag all
set protocols bfd traceoptions file bfd
set protocols bfd traceoptions file size 30m
set protocols bfd traceoptions flag all
|
[Back to Top]
References:
How To
Technical Bulletins
2020-07-16: Command added to VPLS (test PE to PE connectivity over VPLS PW)
2020-05-12: Updated VPLS CLI Commands
2020-04-30: Updated SNMP CLI Commands
2017-04-24: Added PTX product series, and did minor edits.