Data Collection can help with issue investigation as well as reduce time to resolve. Each problem/issue could require a different set of data to collect. This article contains a list of data to collect for SBR Carrier issues.  Should your issue have to be sent to engineering to perform a Root Cause Analysis (RCA), this data is required.

What information should I collect to assist in troubleshooting prior to opening a case?

The goal of this document is to reduce the time spent on initial data collection and reduce time to resolve by providing a comprehensive list of what to collect or gather to troubleshoot an issue.
 

Please collect the following for 'SBR Carrier' Series issues:

Main Server Logs

In the majority of cases, the main server log is the most important piece of data to collect. The higher the logging levels (LogLevel, TraceLevel), the better, with the caveat that on a busy server or under-powered hardware full debug logging can impact performance.

The entire log file is preferred, rather than just a screen shot or log snippet. The log file that contains the messages written while the problem is occurring is most helpful, but it is often helpful to have also the log file of the previous day if it contains messages leading up to the problem.

The SBR server log files (YYYYMMDD.log) are typically found in the installation directory, /opt/JNPRsbr/radius, unless otherwise specified using the LogDir parameter in radius.ini.   The log file name format might differ slightly depending on the log rollover settings (size, time, etc), but will typically start with the date (YYYYMMDD format) and end with ".log".
 

Collect Configuration and SBR Installation Specific Data

The SBR Information Report (SIR.sh) script can be used to collect configuration, system libraries, dictionaries and scripts that will assist in reproducing SBR Carrier issues.  Refer to: KB33206 - Generating a Level1 SIR.sh data collection to share with JTAC
 

Additional Data

Aside from the Main Server Logs and SIR Level1 data, other data might be requested on a case-by-case basis. The following is a list of data and commands for several problem types that would be helpful to have submitted with a new case, in order to quickly begin the RCA (root cause analysis) process:

• SBR crashed and produced a core file
• Radius process memory allocation failures or suspected memory leak
• SBR appears to be hung, unresponsive
• Proxy problem
• Problem with ndbcluster
• simauth problem
• Can't launch SBR Administrator
• Can't launch SBR Web GUI

 

---

SBR crashed and produced a core file

1.  Main Server Logs.
2.  Utilize SIR Level2 functionality to collect core and library specific data

   # cd /opt/JNPRsbr/radius/install/debug
# vi SIR.conf
   Edit Level_Two: Enable = 1
   Under [Core_Files]
   Specify Core file: example, SBR Core File, set radius =1
   Save file
   # ./SIR.sh -d <Destination to save SIR output> -r <Explicit directory location of core file>
   Collect SIR tar ball (where user configure 'Destination Directory' during SIR.sh execution) and send to JTAC
    

---

Radius process memory allocation failures or suspected memory leak

1. Main Server Logs. 
2. A core file generated by gcore (with libumem debug mode on Solaris), preferably when process is larger than 1 Gb and smaller than 3 Gb. Preferably 2 core files, showing some significant file growth between them.

   NOTE: libumem debug is enabled in sbrd.conf by uncommenting the lines for UMEM_DEBUG and UMEM_LOGGING (RADIUS_LD_PRELOAD already uncommented) (restart required)
   # gcore -o <output core filename> <radius process ID>

3. Utilize SIR Level2 functionality to collect gcore data (see SBR crashed and produced a core file)

 

---

SBR appears to be hung, unresponsive

1. Main Server Logs.
2. A core file generated by gcore.

   # gcore -o <output core filename> <radius process ID>

3. Utilize SIR Level2 functionality to collect gcore data (see SBR crashed and produced a core file)

 

---

Proxy problem

1. Main Server Logs.
2. Packet capture. Unfiltered packet capture showing traffic to and from SBR, from both Radius clients and proxy targets.  Preferably taken on the SBR Carrier system, using tcpdump or snoop.
3. Files: radius.ini, proxy.ini, *.pro files, filter.ini, and proxyrl.ini

 

---

Problem with ndbcluster

1. Main Server Logs.
2. Cluster status.

   # ./sbrd status
hadm> ndb_mgm -e show (for older versions where sbrd status does not include the ndb status)

3. ndb_error_reporter logs.  Run as hadm user on a 'M' (or 'SM') node. /opt/JNPRmysql/install/bin/ndb_error_reporter

   hadm> ./ndb_error_reporter config.ini hadm (or hadm username)

 

---

simauth problem

1. Main Server Logs
2. Config files: simauth.aut, gsmmap.gen, GWrelay.conf, conf/authGateway.conf, and conf/ulcmmg.conf
3. Omnimon files output.

   # /opt/ulcm/bin/omnimon

 

---

Can't launch SBR Administrator

(SBR Carrier 7.x)

1. Unfiltered packet capture (snoop, tcpdump) of the connection attempt from either/both of the client and the server.
2. SBR server status

   # ./sbrd status

3. netstat output (server side)

   # netstat -an

4. pfiles output (server side)

   # pfiles (radius pid)

 

---

Can't launch SBR Web GUI

(SBR Carrier 8.x)

1. Packet capture (snoop, tcpdump) on TCP port 2909 between client and SBR.
2. SBR server status

   # ./sbrd status -v
Ensure there is a 'webserver' process

3. netstat output (server side)

   # netstat -an | grep 2909

4. Collect webserver.log for JTAC

   # /opt/JNPRsbr/radius/website/webserver/jetty/logs

2019-04-02: updated collection information, added link to KB33206