Getting Stack Traces -- Detailed Version
This page provides detailed information on obtaining stack traces with debugging symbols. GettingTraces should be enough information for most people to get the extra information, so this page is here for those interested in extra background information or that have more complicated cases. To obtain a stack trace with debugging symbols, you need to do the following:
Install debugging information packages from your distro
Learn what a stack trace is (as well as what it isn't), so that you can submit the right information
Obtain a new stack trace and submit it.
Installing debugging information packages
You need to install packages with debugging information for the program that crashed plus its major dependencies. The steps to do so vary by distro, so you'll need to read the section of GettingTraces/DistroSpecificInstructions corresponding to your distro to find out how.
Note that if you cannot figure out which packages to install (the GettingTraces/DistroSpecificInstructions page merely says how to install them in general, not which ones to install), there are a few pointers to help you:
- Installing extra packages doesn't hurt (other than taking up a little extra disk space)
- Installing the debugging packages for the program that crashed plus all its dependencies is a pretty good rule of thumb in practice, even if you don't really need the debug packages for every dependency.
The libraries that the stack trace shows being used (see below) provide the best hint at determining the exact packages needed. Find out which package owns the library (e.g. 'rpm -qf <library name>') and then look for a debugging information package with a similar name.
- You are welcome to reopen your bug report and ask for which packages to install, if the above steps don't work or you can't figure it out
- If someone else tells you which packages to install, they may use different names than the exact ones you need. This is because distros might rename modules or split them up, meaning that you may have to translate 'glib' to 'libglib2-dbg' or 'gtk+' to 'gtk2-debuginfo').
Learning what a stack trace isn't as well as what it is
Things to avoid; what a stack trace isn't
First of all, let's cover a couple quick things that a stack trace is not. Please do not submit any of these if you are asked for a stack trace:
The output of strace. strace gives a system call trace, not a stack trace.
The output of gdb starting up. Lines beginning with things like "Loading symbols", "Reading symbols", or "(no debugging symbols found)" are not useful.
What a stack trace is
A stack trace is a list of function calls that leads to some point in the program (which happen to be listed in reverse order). Stack traces without debugging symbols typically consist of several lines (also known as frames) with the format
#<frame number> <random memory address> in <function name> () from <library name>
while stack traces with debugging symbols consist of several lines in the format
#<frame number> <random memory address> in <function name> (<variables and their values>) at <filename>:<linenumber>
Note that both kinds of lines can be mixed in a single stack trace when debugging information is only available for some of the relevant libraries. While stack traces without debugging symbols can sometimes be useful to developers, most often developers will need stack traces with debugging symbols to be able to do anything.
An example of a stack trace without debugging symbols is:
[New Thread -1227430208 (LWP 5459)] 0xb721c071 in __waitpid_nocancel () from /lib/tls/libpthread.so.0 #0 0xb721c071 in __waitpid_nocancel () from /lib/tls/libpthread.so.0 #1 0xb7ee01a8 in libgnomeui_module_info_get () from /usr/lib/libgnomeui-2.so.0 #2 <signal handler called> #3 0xb7f7728a in image_loader_start () from /usr/lib/libgthumb-2.6.9.so #4 0xb756223e in g_main_context_is_owner () from /usr/lib/libglib-2.0.so.0 #5 0x080e99c0 in ?? () #6 0xb75be218 in ?? () from /usr/lib/libglib-2.0.so.0 #7 0x08383c80 in ?? () #8 0xb75beaa4 in g_idle_funcs () from /usr/lib/libglib-2.0.so.0 #9 0xb756017a in g_source_is_destroyed () from /usr/lib/libglib-2.0.so.0 #10 0x08383c80 in ?? () #11 0xb7f77270 in image_loader_start () from /usr/lib/libgthumb-2.6.9.so . . .
Note the lines with "??" in them; those make a stack trace far less useful. Also, for the curious, the most useful frames in a stack trace are the ones immediately following the "<signal handler called>" line (since, remembering that functions are listed in reverse order, these lines correspond to the function calls immediately before the crash).
A stack trace with debugging symbols will look similar to the following:
[New Thread -1228469328 (LWP 7113)]
0xffffe410 in __kernel_vsyscall ()
#0 0xffffe410 in __kernel_vsyscall ()
#1 0xb770548b in __waitpid_nocancel ()
from /lib/tls/i686/cmov/libpthread.so.0
#2 0xb7e048e6 in libgnomeui_module_info_get ()
from /usr/lib/libgnomeui-2.so.0
#3 <signal handler called>
#4 IA__g_str_hash (v=0x0) at gstring.c:95
#5 0xb768e73f in IA__g_hash_table_lookup (hash_table=0x93ba780, key=0x0)
at ghash.c:242
#6 0x080f0620 in get_link_files (target_file=<value optimized out>)
at nautilus-file.c:1355
#7 0x080f0713 in nautilus_file_emit_changed (file=0x8d90080)
at nautilus-file.c:5396
#8 0x080dcaea in nautilus_directory_emit_change_signals (directory=0x0,
changed_files=0x940b8b0) at nautilus-directory.c:762
.
.
.In this stack trace, frames 0-2 don't actually have debugging symbols, but only the frames after the "<signal handler called>" are relevant.
Also worth noting is Alex Larsson's guide on The art of decoding backtraces without debug info which provides some directions on how to make a stack trace without debugging symbols more useful. It's a huge amount of work and can only help a little bit at most, but if it's absolutely impossible to get a stack trace with debugging symbols from reporters, developers might find this handy.
Obtaining a stack trace
There are a few different ways to obtain a stack trace with bug-buddy and gdb (you only need to do one of these):
Obtaining a stack trace using only Bug Buddy (easiest)
Obtaining a stack trace using GDB for a program that is not yet running
Obtaining a stack trace using GDB for a program that is already running
Obtaining a stack trace for an applet using just GDB
Obtaining a stack trace using only Bug Buddy
If you use the GNOME Bug Report Tool, bug-buddy, to submit bugs then a stack trace is usually included automatically (unfortunately, sometimes Bug Buddy isn't able to obtain one). For those on the most common linux distributions, you should be able to install Bug Buddy from your installation CDs that came with your distro. All you need to do is make the program crash again and then Bug Buddy should pop up.
Obtaining a stack trace using GDB
First, run the following command to start gdb:
gdb name-of-program
Where name-of-program is the name of the program that you know how to crash (for example: /usr/bin/gnome-panel).
Then, at the gdb prompt, type:
run
Once the program is running, reproduce the crash and go back to the terminal where you ran gdb. The gdb prompt should be shown - if not, hit Control+C to break into the debugger. At the gdb debugger prompt, type:
thread apply all bt
If you see the line
---Type <return> to continue, or q <return> to quit---
just hit each time the Return/Enter key until you see the gdb prompt again. The output is the stack trace. Cut and paste all of it into a text file.
If that does not work (meaning that you don't get any output--this may be the case in programs which are not multi-threaded), type bt instead. If you still do not have any output, read this note about obtaining a stack trace under special circumstances.
You can quit gdb by typing quit.
Obtaining a stack trace using GDB with a running program
First, you need to know the name of the program that is running (e.g. 'nautilus' for the file manager, 'gnome-panel' for the taskbar, etc.). Now, while the program is running, open a terminal and type
ps aux | grep name-of-program | grep -v grep
The second column of the output provides the PID (process ID) of the running program. Now run
gdb $(which name-of-program) PID
substituting for name-of-program and PID appropriately (e.g. gdb $(which nautilus) 6732). This will attach gdb to the running program. Then, at the gdb prompt, type:
cont
to allow the program to continue running. Make the program crash, then go back to the gdb prompt and type
thread apply all bt
If you see the line
---Type <return> to continue, or q <return> to quit---
just hit each time the Return/Enter key until you see the gdb prompt again. The output is the stack trace. Cut and paste all of it into a text file.
If that does not work (meaning that you don't get any output--this may be the case in programs which are not multi-threaded), type bt instead. If you still do not have any output, read this note] about obtaining a stack trace under special circumstances.
You can quit gdb by typing quit.
Obtaining a stack trace for an applet using just GDB
First, you need to make sure that you don't have this applet running:
killall name-of-applet-program
If you had the applet on one of your panel, a dialog will ask you if you want to reload the applet. Ignore this dialog for the moment. Run this command to start gdb:
gdb name-of-applet-program
Note that the applet program is problably not in your path and it usually lives in /usr/libexec.
Then, at the gdb prompt, type:
run
If the program immediately returns, then it means that the applet is already running and you did not kill it. Try to kill it again and type run again.
Once the program is running, return to the dialog that asked you to reload the applet and accept to reload the applet. If you did not have this dialog, then simply add the applet to the panel.
Then reproduce the crash and go back to the terminal where you ran gdb. The gdb prompt should be shown - if not, hit Control+C to break into the debugger. At the gdb debugger prompt, type:
thread apply all bt
If that does not work (meaning that you don't get any output--this may be the case in programs which are not multi-threaded), type bt instead. If you still do not have any output, read this note] about obtaining a stack trace under special circumstances. The output is the stack trace. Cut and paste all of it into a text file.
You can quit gdb by typing quit.
DavydMadeley has some notes on debugging GNOME applets here.
Obtaining stack traces under special circumstances
If you do not get any output from gdb after typing thread apply all bt or bt, it may be because the program is run as root or as another user. In GNOME, this is the case when running some games. In such cases, you will need to be root in order to capture a trace. So, quit gdb, login as root, and then repeat the steps to obtain a stack trace. Some applications may require special treatment in order to debug or may have additional ways of assisting the user or developer to get extra kinds of debugging information. Please examine Bugsquad/TriageGuide/ProductSpecificGuidelines for details.
