Data Collection can help with issue investigation as well as reduce time to resolve. Each problem/issue could require a different set of data to collect. This article contains a list of data to collect for SBR Carrier issues. Should your issue have to be sent to engineering to perform a Root Cause Analysis (RCA), this data is required.
What information should I collect to assist in troubleshooting prior to opening a case?
The goal of this document is to reduce the time spent on initial data collection and reduce time to resolve by providing a comprehensive list of what to collect or gather to troubleshoot an issue.
Please collect the following for 'SBR Carrier' Series issues:
Main Server Logs
In the majority of cases, the main server log is the most important piece of data to collect. The higher the logging levels (LogLevel, TraceLevel), the better, with the caveat that on a busy server or under-powered hardware full debug logging can impact performance.
The entire log file is preferred, rather than just a screen shot or log snippet. The log file that contains the messages written while the problem is occurring is most helpful, but it is often helpful to have also the log file of the previous day if it contains messages leading up to the problem.
The SBR server log files (YYYYMMDD.log) are typically found in the installation directory,
/opt/JNPRsbr/radius
, unless otherwise specified using the LogDir parameter in
radius.ini
. The log file name format might differ slightly depending on the log rollover settings (size, time, etc), but will typically start with the date (YYYYMMDD format) and end with ".log".
Collect Configuration and SBR Installation Specific Data
The SBR Information Report (SIR.sh) script can be used to collect configuration, system libraries, dictionaries and scripts that will assist in reproducing SBR Carrier issues. Refer to:
KB33206 - Generating a Level1 SIR.sh data collection to share with JTAC
Additional Data
Aside from the Main Server Logs and SIR Level1 data, other data might be requested on a case-by-case basis. The following is a list of data and commands for several problem types that would be helpful to have submitted with a new case, in order to quickly begin the RCA (root cause analysis) process:
- Main Server Logs.
- Utilize SIR Level2 functionality to collect core and library specific data
# cd /opt/JNPRsbr/radius/install/debug
# vi SIR.conf
Edit Level_Two: Enable = 1
Under [Core_Files]
Specify Core file: example, SBR Core File, set radius =1
Save file
# ./SIR.sh -d <Destination to save SIR output> -r <Explicit directory location of core file>
Collect SIR tar ball (where user configure 'Destination Directory' during SIR.sh execution) and send to JTAC
- Main Server Logs.
- A core file generated by gcore (with libumem debug mode on Solaris), preferably when process is larger than 1 Gb and smaller than 3 Gb. Preferably 2 core files, showing some significant file growth between them.
NOTE: libumem debug is enabled in sbrd.conf by uncommenting the lines for UMEM_DEBUG and UMEM_LOGGING (RADIUS_LD_PRELOAD already uncommented) (restart required)
# gcore -o <output core filename> <radius process ID>
- Utilize SIR Level2 functionality to collect gcore data (see SBR crashed and produced a core file)
- Main Server Logs.
- A core file generated by gcore.
# gcore -o <output core filename> <radius process ID>
- Utilize SIR Level2 functionality to collect gcore data (see SBR crashed and produced a core file)
- Main Server Logs.
- Packet capture. Unfiltered packet capture showing traffic to and from SBR, from both Radius clients and proxy targets. Preferably taken on the SBR Carrier system, using tcpdump or snoop.
- Files:
radius.ini
, proxy.ini
, *.pro
files, filter.ini
, and proxyrl.ini
- Main Server Logs.
- Cluster status.
# ./sbrd status
hadm> ndb_mgm -e show
(for older versions where sbrd status does not include the ndb status)
- ndb_error_reporter logs. Run as hadm user on a 'M' (or 'SM') node. /opt/JNPRmysql/install/bin/ndb_error_reporter
hadm> ./ndb_error_reporter config.ini hadm (or hadm username)
- Main Server Logs
- Config files:
simauth.aut
, gsmmap.gen
, GWrelay.conf
, conf/authGateway.conf
, and conf/ulcmmg.conf
- Omnimon files output.
# /opt/ulcm/bin/omnimon
(SBR Carrier 7.x)
- Unfiltered packet capture (snoop, tcpdump) of the connection attempt from either/both of the client and the server.
- SBR server status
# ./sbrd status
- netstat output (server side)
# netstat -an
- pfiles output (server side)
# pfiles (radius pid)
(SBR Carrier 8.x)
- Packet capture (snoop, tcpdump) on TCP port 2909 between client and SBR.
- SBR server status
# ./sbrd status -v
Ensure there is a 'webserver' process
- netstat output (server side)
# netstat -an | grep 2909
- Collect webserver.log for JTAC
# /opt/JNPRsbr/radius/website/webserver/jetty/logs