Stack Frame Navigation and Memory Inspection

When analyzing a crashed application, navigating the call stack is essential. The bt (backtrace) command lists the function calls leading to the current point. To inspect variables within a specific stack frame, use the frame command followed by the frame index.

(gdb) bt
#0  AsyncWorker::Execute (this=0x1045dfe0) at worker.cpp:45
#1  0x0000003fe8a2dbe0 in ThreadManager::InternalRun (ctx=0x1045dfe0) at thread_mgr.cpp:82
#2  0x0000003fe8a2d5c4 in thread_start (t=0x1045dfe0) at os_thread.cpp:90
#3  0x0000003fe741bd94 in start_thread (arg=0x3fff920ff050) at pthread_create.c:336
(gdb) frame 0
#0  AsyncWorker::Execute (this=0x1045dfe0) at worker.cpp:45
45	    if (currentStatus != previousStatus &&

Inspecting memory buffers requires pointer dereferencing combined with a length specifier using the @ operator.

(gdb) p *byteStream@8
$1 = {0x41, 0x0, 0x62, 0x0, 0x63, 0x0, 0x0, 0x0}

For null-terminated strings, printing the pointer directly is usually sufficient. However, if the output is truncated, you must adjust the print elements setting to display more characters.

(gdb) show print elements 
Limit on string chars or array elements to print is 200.
(gdb) set print elements 2048
(gdb) show print elements 
Limit on string chars or array elements to print is 2048.
(gdb) p errorMessage.toStdString().c_str()
$2 = 0x3fff65003960 "Error: Connection timed out in module A"

Initiating a Debugging Session

To begin debugging, launch the GNU Debugger (GDB) followed by the executable name. Once inside the GDB shell, use the run command to start the program. You can pass arguments directly to the target application after the run command.

[root@server ~]# gdb order_processor 
GNU gdb (GDB) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
Reading symbols from order_processor...done.
(gdb) run --config=/etc/production.cfg
Starting program: /opt/local/bin/order_processor --config=/etc/production.cfg
---Loading debug symbols---

If the program is already running, you can interrupt execution by pressing Ctrl+C. This pauses the process, allowing you to set breakpoints, inspect the stack, or examine variables. Use the continue command to resume execution.

Processing batch ID 5502
^C
Program received signal SIGINT, Interrupt.
0x00007ffff7b2d35c in __read () at ../sysdeps/unix/syscall-template.S:84
84	T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) b ProcessOrder
Breakpoint 1 at 0x4005e2: file main.cpp, line 58.
(gdb) c
Continuing.

Attaching to Running Processes and Stack Sampling

For troubleshooting issues in production environments, you can attach GDB to an already running process using its Process ID (PID).

[root@server ~]# pidof order_processor
30215
[root@server ~]# gdb -p 30215
GNU gdb (GDB) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
Attaching to process 30215
Reading symbols from /usr/lib/libssl.so.1.1...done.
Reading symbols from /usr/lib/libcrypto.so.1.1...done.
0x00007ffff7b2c35c in __read () at ../sysdeps/unix/syscall-template.S:84
84	T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb)

Upon attachment, the process pauses. You should set breakpoints at critical locations, such as error handling routines or specific logic branches, and then resume execution with continue. If a generic error handler is called from multiple locations, setting a breakpoint there allows you to inspect the backtrace to determine the exact call path.

(gdb) b HandleError
Breakpoint 1 at 0x401150: file utils.cpp, line 22.
(gdb) b main.cpp:195
Breakpoint 2 at 0x4008a0: file main.cpp, line 195.
(gdb) c
Continuing.

Breakpoint 2, main (argc=1, argv=0x7fffffffe518) at main.cpp:195
195				std::cin >> userInput;
(gdb) bt
#0  main (argc=1, argv=0x7fffffffe518) at main.cpp:195
(gdb)

Alternatively, for a lightweight inspection without stopping the application, use gstack (a shell script wrapper around GDB) to sample the stack traces of all threads.

[root@server ~]# pidof order_processor
30215
[root@server ~]# gstack 30215
Thread 5 (Thread 0x7ffff6ffb700 (LWP 30220)):
#0  0x00007ffff7bc8670 in pthread_cond_wait (cond=0x6052d8, mutex=0x6052b0) at pthread_cond_wait.c:186
#1  0x0000000000401d3c in ThreadPool::WorkerLoop() () from /usr/local/lib/libcommon.so

You can redirect the output of gstack to files and compare them to analyze thread behavior over time, which is useful for detecting deadlocks or performance bottlenecks.

[root@server ~]# gstack 30215 > snapshot_a.txt
[root@server ~]# sleep 5
[root@server ~]# gstack 30215 > snapshot_b.txt
[root@server ~]# diff snapshot_a.txt snapshot_b.txt