Support Support Downloads Knowledge Base Juniper Support Portal Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Contrail] Monitoring real-time backtrace with live gdb using shell script

0

0

Article ID: KB35082 KB Last Updated: 26 Sep 2019Version: 1.0
Summary:

vrouter coredump files contain the address space (memory) of a vrouter agent process when it terminates unexpectedly, and are usually triggered by the kernel in response to vrouter agent crashes. From the coredump files, experienced engineers/developers can find important clues to the problem's cause because these files retain a post-mortem snapshot of the vrouter agent's state at the time of the crash, and come up with fixes.

One problem that is faced is reproducing the exact coredump file in the lab environment so that developers/engineers can look at the running setup and explore possible triggers for the problem. Unfortunately, it is usually difficult to trigger the same coredump.

In this article, we introduce a method that uses a script to dump a backtrace of all threads of a running vrouter agent process with gdb, so that the backtrace can be monitored and collected in real time.

Solution:

Even without a coredump generated in the lab test, we can still run gdb on the running vrouter agent process to collect the backtrace in real time:

gdb --batch -ex "thread apply all bt" -p `pidof contrail-vrouter-agent` >> logfile.log
  • The --batch option instructs gdb to run in "batch mode."

  • The -ex option gives commands that you typically run under gdb, for example "bt," "bt all," "thread 2," and so on. Here we give thread apply all bt to retrieve the backtrace of all threads simultaneously.

  • The -p option gives the process ID. Here we use pidof to retrieve the pid from the process name, which is contrail-vrouter-agent in Contrail compute.

With this "batch" mode gdb command, real-time backtraces of all threads of the running vrouter agent process can be collected in a file. With the following one-liner shell script, we can keep collecting data when the test is ongoing:

while :; do date; echo "pass 40s"; \
gdb --batch -ex "thread apply all bt" -p `pidof contrail-vrouter-agent` >> logfile.log\
sleep 30; done

Developers/engineers can then check the logfile.log anytime for any suspicious data. 

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search