[ScreenOS] Data Collection Checklist - Logs/data to collect for troubleshooting

Section Contents

Contents, Caveats, and Tips
Data to Collect for All Configurations
Data to Collect for Specific Issues
Reference Links

Caveats and Tips

Debugs require additional system resources to gather and store data
- Dbuf stream is limited to a maximum of 4 MB. On SSG series devices, the debug output can be saved to a USB flash drive.
- Gauge current system utilization before enabling any debugs.
  - get performance cpu all detail
- Instead of using "all”, you can flag specific areas of interest.
- Don’t forget to turn off all debugs before exiting.
  - "undebug all" or press the escape key (Esc) on your keyboard.
  - Any administrator logging out of the system will automatically turn off all debugs.
To deactivate paging (output stopping each page requiring you to press the space bar), issue set console page 0.
- To re-enable, issue unset console page or set console page <number of lines>.
All output can be redirected to a TFTP server by appending "> tftp x.x.x.x filename".
- x.x.x.x is the IP of the TFTP server.
- filename is the name of the file to which the output is saved on the TFTP server.
Before collecting ANY information, ensure that logging is enabled on your terminal application (Putty, SecureCRT, etc.). Some of the outputs can be very long.
<> indicates that a variable must be entered (for example, <x.x.x.x> must be replaced with the IP address in question).
# indicates a comment. Read the comments. You can paste the comments into the command line without any issues.
All debugs indicated need to be run at the same time. For example, Anti-Spam requires both debug flow basic and debug anti-spam all at the same time.

Data to Collect for All Configurations

Regardless of configuration, all cases will benefit by attaching the session captures, requested information output, and logs when initially opening the case. If you need to investigate an intermittent concern (for example, slow transfers at peak hours), be sure to collect this data at the time of the problem.

Basic information to collect for all issues	Background Information	1. Provide all SSH/Telnet/console session captures 2. Provide any available topology information 3. Summary of how the device is being used (production, lab system, co-location, etc.) 4. Summary of device history (new install, production for X months/years, other recent cases, etc.) 5. Summary of any recent changes in the network or on the device
Basic information to collect for all issues	Logs	`get tech-support get event get log sys get log sys saved get performance cpu all detail get performance session detail get session info get dbuf stream`

See Reference section for the following:

How to gather the data
Resolution Guides
Technical Bulletins
Technical Documentation

Data to Collect for Specific Issues:

Anti-Spam
Antivirus (AV)
Application Layer Gateway (ALG)
Asymmetric Digital Subscriber Line (ADSL)
Authentication
Auto Connect VPN (AC-VPN)
Border Gateway Protocol (BGP)
Buffer (net-pak) leak
Crash
Deep Inspection (DI)
Dial-up VPN
Dynamic IP (DIP)
Extended Authentication (XAuth)
Frame Relay
GPRS Tunneling Protocol (GTP)
H.323
High CPU (5.4 and lower)
High CPU (6.0r2 and higher)
Infranet Controller / Enforcer
Integrated Services Digital Network (ISDN)
Management
Media Gateway Control Protocol (MGCP)
Memory
Multicast (IGMP)
Multicast (PIM)
Multi-Link Frame Relay
NetScreen Redundancy Protocol (NSRP)
Open Shortest Path First (OSPF)
Point to Point Protocol (PPP) (including Multi-link PPP)
Point to Point Tunneling Protocol (PPTP)
Policy Based Routing (PBR) Port Authentication (802.1x)
Remote Procedure Call (RPC)
Session Initiation Protocol (SIP)
Skinny Client Control Protocol (SCCP)
Static Routing
SurfControl (Redirect or integrated URL filtering)
Throughput
Traffic failing for a specific host / application -- (Basic debug)
Traffic Shaping
Virtual Private Network (VPN)
Virtual Router Redundancy Protocol (VRRP)
WebSense (Redirect URL filtering)
Wireless

Anti-Spam	Commands	`get memory get os task get net-pak stats get socket get tcp get anti-spam`
	Debugs	`debugs from` "Traffic failing for a specific host/application" `debug anti-spam all`
	Additional Information	External sniffer captures from the directly connected device on both sides of the firewall.

Antivirus (AV)	Commands	`get av get av all get av scan get av stat get av session get asp stat get memory get os task get net-pak stats get socket get tcp get av scan-mgr` Repeat next block of commands 4 to 5 times at 10-second intervals: `get clock` `get counter stat get net-pak stats get mem get os task get session info get net-pak`
Antivirus (AV)	Debugs	`debug apppry basic debug av basic debug av engine` If update related: `debug av updater debug httpfx all` If scan related`: debug av scan-file debug av no-scan`

Application Layer Gateway (ALG)	Commands	`get alg get alg <alg name> get rm group active get rm resource active get nat cookie \| include allocated`
	Debugs	`debug <alg name> all` # If it fails, then "`debug nat <alg name>`" `debug rm all debug nat gate debugs from`"Traffic failing for a specific host/application"
	Additional Information	Try disabling ALG globally using `unset alg <alg name>`, or per policy using the commands below: `set policy id <#> set application ignore`

Asymmetric Digital Subscriber Line (ADSL)	Commands	`Get adsl basic` `Get int ad` `Get adsl stat` `Get adsl <#> basic` `Get adsl <#> if-stats` `Get adsl <#> sar` `get adsl <#> stats` `get adsl <#> training-status` `get adsl <#> vc-info` `get adsl <#> memory` `get interface adsl` `get interface adsl extensive` If PPPoE: `get pppoe all get pppoe name <name>` If PPPoA: `Get pppoa all Get pppoa name <name>`
	Debugs	`Exec adsltest <#> ping` If PPPoE: `Debug pppoe basic Exec pppoe name <name> disconnect Clear dbuf Exec pppoe name <name> connect Undebug all Get dbuf` `stream` If PPPoA: `Debug pppoa all Exec pppoa name <name> disconnect Clear dbuf Exec pppoa name <name> connect Undebug all Get dbuf stream`
	Additional Information	See KB12137 – "ADSL card is not loading; event log "ADSL2/0 SOC Firmware Failed (Load Bootrom Failure)" is reported"

Authentication	Commands	`get auth history` `get auth queue` `get auth statistics` `get auth settings` `get auth table` If L2tp: `get L2tp all get L2tp <name> get L2tp <name> active`
Authentication	Debugs	`debug auth <option> Debug proxy all Debugs from`"Traffic failing for a specific host/application"

Auto Connect VPN (AC-VPN)	Commands	Run the following commands on the hub and both spokes simultaneously: `get ike cookie get nsp get sa get sa stat get sa id <#>` # SA IDs are in hex, so you need to put "0x" in front of the id. # Example: get sa id 0x01 Enter the virtual router configured for NHRP using `set vr <vr>`. `get protocol nhrp get proto nhrp cache get proto nhrp peer`
Auto Connect VPN (AC-VPN)	Debugs	Run the following debugs on the hub and both spokes simultaneously: `set sa-filter <x.x.x.x>` # <x.x.x.x> is the peer gateway. `debug ike detail debug pki detail` # Use PKI debug only if VPN uses certificates. `debug auth all debug nhrp all`

Border Gateway Protocol (BGP)	Commands	`get route get route summary get session src-ip <x.x.x.x> get session dst-ip <x.x.x.x> get route ip <x.x.x.x>` # <x.x.x.x> is the route in question. `get route id <#>` # <#> is the ID of the route in question and is found in the "get route" or "get route ip" command. `get os task get task bgp get interface <interface> \| include "status change"` # Available only in 6.2 and 6.3. Enter the virtual router configured for BGP using `set vr <vr>`. `get proto bgp config get proto bgp neighbor get proto bgp network get proto bgp redistrib get proto bgp rib-in get proto bgp router-id get proto bgp rib received` # rib received available only in 6.2 and 6.3. `get proto bgp rib advertised` # Available only in 6.2 and 6.3.
Border Gateway Protocol (BGP)	Debugs	`debug bgp all`

Buffer (net-pak) leak	Commands	`get session frag get tcp` Repeat next block of commands 2 to 3 times at 10-second intervals: `get clock get performance cpu all detail get performance session detail get net-pak get net-pak stats get net-pak distribute get net-b get memory pool get counter statistics`
Buffer (net-pak) leak	Debugs	If ‘failed’ counter from ‘get net-pak stats’ is incrementing and the ‘free’ counters do not decrement for over one hour, collect the following information. `debug npak get` # Keep this running and ensure no one logs in to or out of the firewall. See "Keep debugs running after logging out" below. Using a script, capture the following command every 15 minutes: `get clock get net-pak stats get net-pak`

Crash	Commands	- Console output of the crash dump: If console output is unavailable, use "get log sys saved"
Crash	Additional Information	- Date and time of crash - Description of the last thing that happened before the crash - Were there any changes in the network before or during the time of the crash (physical or configuration)? If so, what changes (added new equipment, links, new interface, etc.)? - What kind of traffic was going through the device at the time of crash? - How long was the firewall running before the incident occurred? - Was anyone logged in to the firewall during the time of the crash (NSM, Telnet, SSH, console, WebUI)? - Has the firewall crashed before?

Deep Inspection (DI)	Commands	`get memory get os task get net-pak stats get socket get tcp`
	Debugs	`debugs from` "Traffic failing for a specific host/application" `debug idp`
	Additional Information	External sniffer captures from the directly connected device on both sides of the firewall.

Dial-up VPN	Commands	`get ike cookie get sa`
	Debugs	If VPN is down, use the following debugs: `set sa-filter <x.x.x.x>` # <x.x.x.x> is the peer gateway. `debug ike detail debug pki detail` # Use PKI debug only if VPN uses certificates. `debug auth all` If authentication is failing, use the following commands: `debug auth all debug proxy all clear dbuf clear auth table` # <make first connection> `get dbuf stream get auth table get auth table id <#> clear dbuf` # <make second connection attempt> `get dbuf stream get auth table get auth table id <#>` If VPN is up, but not passing traffic: `set sa-filter <remote gateway IP> debug ike basic debug ike natt debugs from`"Traffic failing for a specific host/application"
	Additional Information	See VPN resolution guide

Frame Relay	Commands	"x" is the slot number, "y" is the port number, and "z" is the subinterface number(tagged interface). `get int serial<x/y> fr` `get int serial<x/y.z> fr` `get int serial<x/y> fr stat` `get int serial<x/y> fr pvc`
	Debugs	`Debug fr lmi Debug fr pkt`
	Additional Information	Provide full network topology, including IP addresses.

High CPU (6.0r2 and higher)	Commands	`set alarm snapshot CPU on set alarm snapshot CPU trigger` # Repeat 2-3 times at 10 second intervals `unset alarm snapshot CPU on get alarm snapshot CPU all set fprofile packet en` Run the following commands three times: `set fprofile vector en set fprofile packet start` # <wait for a few seconds> `set fprofile packet stop get fprofile packet get fprofile packet ip get fprofile packet ip dport get fprofile packet ip dst-ip get fprofile packet ip sport get fprofile packet ip src-ip get fprofile packet ip proto get fprofile packet none-ip get fprofile packet none-ip dst-mac get fprofile packet none-ip proto get fprofile packet none-ip src-mac get fprofile vector` After complete, run "unset fprofile packet enable". Also capture the information listed in "High CPU (5.4 and lower)"
	Debugs	Debugs from "Traffic failing for a specific host/application" # Do not set the flow filters (make sure "get ff" returns nothing). For ISG series, collect the following as well. Repeat next block of commands 4 to 5 times at 10-second intervals: `get sat 0 d` `get asic d` For 5200 / 5400, collect the following as well. Repeat next block of commands 4 to 5 times at 10-second intervals: <X> is 0 to 5, or until you receive "out-of-range" message `get sat <X> d`
	Additional Information	See KB8910 - Determining Which Task is Using Most Resources on the CPU For those cases where there is a 1-second spike in CPU, it’s difficult to run the "trigger" at that moment – especially for TASK CPU. Instead, change the CPU alarm threshold so that it is triggered automatically. As soon as the spike is seen, a snapshot is automatically triggered and written to the buffer. No other external script is needed. You do not even need to be logged in to the firewall at the time of the spike. `set alarm threshold cpu 70 set alarm snapshot cpu on # <let firewall run until spike is seen> # the firewall can be left unattended get alarm snapshot cpu all`

Dynamic IP (DIP)	Commands	`Get dip Get dip all Get interface <interface> dip Get interface <interface> dip detail Get pport` # Only if using interface-NAT.
Dynamic IP (DIP)	Debugs	`Debug dip all` `Debug nat gate` `Debug nat xlate` `Debug nat pport` # Debug nat pport only if using interface-NAT. `Debugs from` "Traffic failing for a specific host/application"

Extended Authentication (XAuth)	Commands	`Get xauth active Get xauth client Get xauth server`
Extended Authentication (XAuth)	Debugs	`debug auth <option> Debugs from`"Traffic failing for a specific host/application" `Debug pptp all` # Only if using pptp.

GPRS Tunneling Protocol (GTP)	Commands	`Get gtp Get gtp tunnel Get ip_action`
	Debugs	`Debugs from` "Traffic failing for a specific host/application" `Debug flow drop Debug gtp all Debug nsgp all`
	Additional Information	Is this GTP inspection or passthrough? Provide external packet captures from both sides of the firewall at the same time as the debugs.

H.323	Commands	`get alg h323 get alg h323 counters get rm group active get rm resource active get nat cookie \| include allocated`
	Debugs	`debug h323 all debug rm all debug nat gate debug nat xlate debugs from`"Traffic failing for a specific host/application"
	Additional Information	Try disabling SIP ALG globally using `unset alg h323`, or per policy using the commands below: `set policy id <#> set application ignore`

High CPU (5.4 and lower)	Commands	`Get gate Get sess frag Get tcp Get dbuf stream get session` # Output can be long. Recommended to redirect to TFTP. `get sess hardware` # Get sess hardware is for the ISG and 5000 series only. # Output can be long. Recommended to redirect to TFTP. Repeat next block of commands 4 to 5 times at 10-second intervals: `get clock get counter stat get perf cpu all detail get perf sess detail get os task get session info` For ISG series, collect the following as well. Repeat next block of commands 4 to 5 times at 10-second intervals: `get clock get sat 0 d get sat 0 x-c get sat 0 fr get sat 0 c get sat 0 s` For 5200 / 5400, collect the following as well. Repeat next block of commands 4 to 5 times at 10-second intervals: <X> is 0 to 5, or until you receive "out-of-range" message. `get clock get sat <X> d get sat <X> x-c get sat <X> fr get sat <X> c get sat <X> s`
	Debugs	Debugs from "Traffic failing for a specific host/application" # Do not set the flow filters (make sure "get ff" returns nothing). If ScreenOS 5.3 or greater: `set alarm snapshot CPU on set alarm snapshot CPU trigger` # Repeat 2-3 times at 10 second intervals `unset alarm snapshot CPU on get alarm snapshot CPU all`
	Additional Information	For those cases where there is a 1-second spike in CPU, it’s difficult to run the "trigger" at that moment – especially for TASK CPU. Instead, change the CPU alarm threshold so that it is triggered automatically. As soon as the spike is seen, a snapshot is automatically triggered and written to the buffer. No other external script is needed. You do not even need to be logged in to the firewall at the time of the spike. `set alarm threshold cpu 70 set alarm snapshot cpu on` # <let firewall run until spike is seen> # the firewall can be left unattended `get alarm snapshot cpu all`

Infranet Controller/ Enforcer	Commands	`Get infranet controller Get infranet enforcer Get auth infranet`
	Debugs	`Debug auth infranet` Debugs from "Traffic failing for a specific host/application"
	Additional Information	See KB11119 – "How to troubleshoot connectivity issues between the Infranet Controller and Infranet Enforcer (Firewall)"

Integrated Services Digital Network (ISDN)	Commands	<x/y> is card number/port number (viewed by "get interface"). `Get int bri<x/y> get int bri<x/y> isdn get int bri<x/y> isdn q921 status get int bri<x/y> isdn q921 status get int bri<x/y> isdn q931 status get int bri<x/y> isdn q931 status get int bri<x/y> extensive get int bri<x/y> bri-options get int bri<x/y> hold-time get int bri<x/y> screen get int dialer<x> get int dialer<x> screen get int dialer<x> ppp get ppp sa get ppp ml get counter flow`
	Debugs	Debugs from "Traffic failing for a specific host/application" `debug isdn all debug ppp basic` To check the Layer 1 status: `debug ipacx basic debug ipacx rx debug ipacx tx debug ipacx phy debug ipacx stack`
	Additional Information	Provide full network topology, including IP addresses.

Management	Commands	`get socket get tcp get memory get os task get net-pak stats get socket get socket id <#>` # <#> is the socket listed in "get socket" for the application in question `get task web get net-buf \| include free get net-pak stats get net-pak distribute`
	Debugs	debugs from`"Traffic failing for a specific host/application" debug socket all debug tcp all debug admin all debug auth all debug web all (if WebUI is the issue) debug httpfx all (if issue with link in WebUI) debug telnet all (if telnet is the issue) debug ssl all (if WebUI via HTTPS is the issue) debug ssh all (is SSH is the issue)`
	Additional Information	Collect the "Commands" before, during, and after the issue occurs.

Media Gateway Control Protocol (MGCP)	Commands	`get alg mgcp get alg mgcp calls get alg mgcp counters get alg mgcp endpoints get alg mgcp sessions`
	Debugs	`debug mgcp all` debugs from "Traffic failing for a specific host/application"
	Additional Information	Try disabling the RPC ALGs globally using `unset alg msrpc` or `unset alg sunrpc`, or per policy using the commands below: `set policy id <#> set application ignore`

Memory	Commands	`Get memory`
	Debugs	Setup: `clear mem debug set mem debug 1000` Run the following commands every 24 hours: `get clock get mem get mem pool get mem ipc get mem chunk get mem ker get mem debug get mem used get os task get net-pak get net-pak s get net-b`
	Additional Information	- Is the memory bar in the WebUI yellow or red? - DO NOT USE "clear mem debug" AFTER TEST HAS STARTED.

Knowledge Base

Data to Collect for All Configurations

Data to Collect for Specific Issues:

References:

Multicast (IGMP)	Commands	`get route get session get igmp group all get igmp interface all get igmp stat`
	Debugs	`debug igmp all debug vr mroute debug vr cachemiss debug vr mroute-lookup debug flow mcast debugs from`"Traffic failing for a specific host/application"
	Additional Information	Network topology showing sender, receiver and RP, as well as all IP addresses. If using both IGMP and PIM on the firewall, collect the information listed under PIM and IGMP at the same time.

Multicast (PIM)	Commands	`get route get session` Enter the virtual router configured for PIM using `set vr <vr>`. `get prot pim st get prot pim nei get prot pim mroute get prot pim rp active get prot pim rpf get prot pim join-prune get mcore get mroute`
	Debugs	`debug pim all debug vr mroute debug vr cachemiss debug vr mroute-lookup debug flow mcast debugs from`"Traffic failing for a specific host/application"
	Additional Information	Network topology showing sender, receiver and RP, as well as all IP addresses. If using both IGMP and PIM on the firewall, collect the information listed under PIM and IGMP at the same time.

Multi-link Frame Relay	Commands	`Get int mlx mlfr_uni_uni Get int serial<x/y> mlfr_uni_uni`
	Debugs	`Debug ml pkt Debug ml fr Debug ml frag Debug ml assem Debug fr lip`
	Additional Information	Provide full network topology, including IP addresses. Verify the frame relay is OK from the frame-relay tab in the WebUI.

NetScreen Redundancy Protocol (NSRP)	Commands	`Get nsrp Get nsrp monitor Get interface Exec nsrp sync global-config check-sum` # If running 6.1 or lower, run "get log sys \| include config" after this command. Repeat the next block of commands 4 to 5 times at 10-second intervals: `Get nsrp Get nsrp counter Get nsrp counter packet Get interface`
	Debugs	`Debug nsrp all`
	Additional Information	Debugs and get commands have to be collected from both cluster members. See NSRP resolution guide.

Open Shortest Path First (OSPF)	Commands	`get route get route summary get session src-ip <x.x.x.x> get session dst-ip <x.x.x.x> get route ip <x.x.x.x>` # <x.x.x.x> is the route in question `get route id <#>` # <#> is the ID of the route in question and is found in the "get route" or "get route ip" command `get os task get task ospf get interface <interface> \| include "status change"` # Status change is only available in 6.2 and 6.3 `get interface <interface> protocol ospf` Enter the virtual router configured for OSPF using `set vr <vr>`. `get protocol ospf area get protocol ospf auth get protocol ospf config get proto ospf int get proto ospf neighbor get proto ospf routes get proto ospf data get proto ospf database detail` # Output can be long. Recommend redirecting to TFTP.
Open Shortest Path First (OSPF)	Debugs	`debug ospf all`

Point to Point Protocol (PPP) (including Multi-link PPP)	Commands	`Get int <interface> ppp Get ppp sa Get ppp ml Get counter flow`
	Debugs	`Debug ppp basic`
	Additional Information	Provide full network topology, including IP addresses.

Point to Point Tunneling Protocol (PPTP)	Commands	`get alg pptp get alg pptp counters get alg pptp xlate get rm group active get nat cookie \| include allocated`
	Debugs	If ALG is disabled, debugs from "Traffic failing for a specific host/application" If ALG is enabled, `debug rm all debug nat gate debug nat xlate` debugs from "Traffic failing for a specific host/application"
	Additional Information

Policy Based Routing (PBR)	Commands	`get route get session` For all virtual routers with PBR configured: `get vr <vr> access-list get vr <vr> action-group get vr <vr> config get vr <vr> match-group get vr <vr> policy`
	Debugs	`debugs from` "Traffic failing for a specific host/application" `debug pbr all debug vr lookup debug vr session`
	Additional Information	Provide full network topology, including IP addresses.

Port Authentication (802.1x)	Commands	`get auth history get auth queue get auth statistics get auth settings get auth table get dot1x get dot1x session get dot1x statistics`
Port Authentication (802.1x)	Debugs	`debug auth <option> Debug dot1x all`

Remote Procedure Call (RPC)	Commands	`get service map-sun-rpc get service map-ms-rpc`
	Debugs	debugs from "Traffic failing for a specific host/application" `debug rpc all`
	Additional Information	Try disabling the RPC ALGs globally using `unset alg msrpc` or `unset alg sunrpc`, or per policy using the commands below: `set policy id <#> set application ignore` See KB11951 – "Troubleshoot MSRPC problems on firewalls running ScreenOS" (login required)