Pleasant debugging with GDB and DDD

GDB front ends

By default, GDB provides a terse line-based terminal. You need to explicitly ask to print the source code being debugged, the values of variables, or the current list of breakpoints. There are four ways to customize this interface. Ordered from basic to complicated, they are:

1. Get used to the default behavior. Then you’ll be comfortable on any system with GDB installed. However, this approach does forego some real conveniences.
2. Enable the built-in GDB TUI mode with the -tui command line flag (available since GDB version 7.5). The TUI creates Curses windows for source, registers, commands, etc. It’s easier to trace execution through the code and spot breakpoints than in the default interface.
3. Customize the UI using scripting, sourced from your .gdbinit. Some good examples are projects like gdb-dashboard and gef.
4. Use a graphical front-end that communicates with an “inferior” GDB instance. Front ends either use the GDB machine interface (MI) to communicate, or they screen scrape sessions directly.

In my experiments, the TUI mode (option two) seemed promising, but it has some limitations:

• no persistent window to display variables or the call stack
• no ability to set or clear breakpoints by mouse
• no value inspection with mouse hover
• mouse scroll wheel didn’t work for me on OpenBSD+xterm
• no interactive structure/pointer exploration
• no historical value tracking for variables (aside from GDB’s Linux-only process record and replay)

Ultimately I chose option four, with the Data Display Debugger (DDD). It’s fairly ancient, and requires configuration changes to work at all with recent versions of GDB. However, it has a lot of features delivered in a 3MB binary, with no library dependencies other than a Motif-compatible UI toolkit. DDD can also control GDB sessions remotely over SSH.

Fixing DDD freeze on startup

As a front-end, DDD translates user actions to text commands that it sends to GDB. Newer front-ends use GDB’s unambiguous machine interface (MI), but DDD never got updated for that. It parses the standard text interface, essentially screen scraping GDB’s regular output. This causes some problems, but there are workarounds.

Upon starting DDD, the first serious error you’ll run into is the program locking up with this message:

1

Waiting until GDB gets ready...

The freeze happens because DDD is looking for the prompt (gdb). However, DDD never sees that prompt because it incorrectly changed the prompt at startup. To fix this error, you must explicitly set the prompt and unset the extended-prompt. In ~/.ddd/init include this code:

1
2
3

Ddd*gdbSettings: \
unset extended-prompt\n\
set prompt (gdb) \n

The root of the problem is that during DDD’s first run, it probes all GDB settings, and saves them in to its .ddd/init file for consistency in future runs. It probes by running show settingname for all settings. However, it interprets the results wrong for these settings:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

exec-direction
extended-prompt
filename-display
interactive-mode
max-value-size
mem inaccessible-by-default
mpx bound
record btrace bts
record btrace pt
remote interrupt-sequence
remote system-call-allowed
tdesc

The incorrect detection is especially bad for extended-prompt. GDB reports the value as not set, which DDD interprets – not as the lack of a value – but as text to set for the extended prompt. That text overrides the regular prompt, causing GDB to output not set as its actual prompt.

Honoring gdbinit changes

As mentioned, DDD probes and saves all GDB settings during first launch. While specifying all settings in ~/.ddd/init might make for deterministic behavior on local and remote debugging sessions, it’s inflexible. I want ~/.gdbinit to be the source of truth.

Thus you should:

1. Delete all Ddd*gdbSettings other than the prompt ones above, and
2. Set Ddd*saveOptionsOnExit: off to prevent DDD from putting the values back.

Dark mode

DDD’s default color scheme is a bit glaring. For dark mode in the code window, console, and data display panel, set these resources:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

Ddd*XmText.background:             black
Ddd*XmText.foreground:             white
Ddd*XmTextField.background:        black
Ddd*XmTextField.foreground:        white
Ddd*XmList.background:             black
Ddd*XmList.foreground:             white
Ddd*graph_edit.background:         #333333
Ddd*graph_edit.edgeColor:          red
Ddd*graph_edit.nodeColor:          white
Ddd*graph_edit.gridColor:          white

UTF-8 rendering

By default, DDD uses X core fonts. All its resources, like Ddd*defaultFont, can pick from only those legacy fonts, which don’t properly render UTF-8. For proper rendering, we have to change the Motif rendering table to use the newer FreeType (XFT) fonts. Pick an XFT font you have on your system; I chose Inconsolata:

1
2
3
4

Ddd*renderTable: rt
Ddd*rt*fontType: FONT_IS_XFT
Ddd*rt*fontName: Inconsolata
Ddd*rt*fontSize: 8

The change applies to all UI areas of the program except the data display window. That window comes from an earlier codebase bolted on to DDD, and I don’t know how to change its rendering. AFAICT, you can choose only legacy fonts there, with Ddd*dataFont and Ddd*dataFontSize.

Although international graphemes are garbled in the data display window, you can inspect UTF-8 variables by printing them in the GDB console, or by hovering the mouse over variable names for a tooltip display.

Remote GDB configuration

DDD interacts with GDB through the terminal like a user would, so it can drive debugging sessions over SSH just as easily as local sessions. It also knows how to fetch remote source files, and find remote program PIDs to which GDB can attach. DDD’s default program for running commands on a remote inferior is remsh or rsh, but it can be customized to use SSH:

1

Ddd*rshCommand: ssh -t

In my experience, the -t is needed, or else GDB warnings and errors can appear out of order with the (gdb) prompt, making DDD hang.

To debug a remote GDB over SSH, pass the --host option to DDD. I usually include these command-line options:

1

ddd --debugger gdb --host admin@example.com --no-exec-window

(I specify the remote debugger command as gdb when it differs from my local inferior debugger command of egdb from the OpenBSD devel/gdb port.)

GDB tricks

Useful execution commands

Beyond the basics of run, continue and next, don’t forget some other handy commands.

• finish - execute until the current function returns, and break in caller. Useful if you accidentally go too deep, or if the rest of a function is of no interest.
• until - execute until reaching a later line. You can use this on the last line of a loop to run through the rest of the iterations, break out, and stop.
• start - create a temporary breakpoint on the first line of main() and then run. Starts the program and break right away.
• step vs next - how to remember the difference? Think a flight of “steps” goes downward, “stepping down” into subroutines. Whereas “next” is the next contiguous source line.

Batch mode

GDB can be used non-interactively, with predefined scripts, to create little utility programs. For example, the poor man’s profiler is a technique of calling GDB repeatedly to sample the call stack of a running program. It sends the results to awk to tally where most wall clock time (as opposed to just CPU time) is being spent.

1
2
3
4
5
6
7
8

# show why program.core died

gdb --batch \
  -ex "set print frame-arguments all" \
  -ex "set print pretty on" \
  -ex "set print addr off" \
  -ex "thread apply all bt full" \
  /path/to/program program.core

You can put this incantation (minus the final program and core file paths) into a shell alias (like bt) so you can run it more easily. To test, you can generate a core by running a program and sending it SIGQUIT with Ctrl-\. Adjusting ulimit -c may also be necessary to save cores, depending on your OS.

User-defined commands

GDB allows you to define custom commands that can do arbitrarily complex things. Commands can set breakpoints, display values, and even call to the shell.

Here’s an example that does a few of these things. It traces the system calls made by a single function of interest. The real work happens by shelling out to OpenBSD’s ktrace(1). (An equivalent tracing utility should exist for your operating system.)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

define ktrace
    # if a user presses enter on a blank line, GDB will by default
    # repeat the command, but we don't want that for ktrace

    dont-repeat

    # set a breakpoint for the specified function, and run commands
    # when the breakpoint is hit

    break $arg0
    commands
        # don't echo the commands to the user
        silent

        # set a convenience variable with the result of a C function
        set $tracepid = (int)getpid()

        # eval (GDB 7.2+) interpolates values into a command, and runs it
        eval "set $ktraceout=\"/tmp/ktrace.%d.out\"", $tracepid
        printf "ktrace started: %s\n", $ktraceout
        eval "shell ktrace -a -f %s -p %d", $ktraceout, $tracepid

        printf "\nrun \"ktrace_stop\" to stop tracing\n\n"

        # "finish" continues execution for the duration of the current
        # function, and then breaks
        finish

        # After commands that continue execution, like finish does,

        # more commands here
    end

    # GDB automatically sets $bpnum to the identifier of the created breakpoint
    set $tracebp = $bpnum
end

define ktrace_stop
    dont-repeat

    # consult $ktraceout and $tracebp set by ktrace earlier

    eval "shell ktrace -c -f %s", $ktraceout
    del $tracebp
    printf "ktrace stopped for %s\n", $ktraceout
end

Hooks

GDB treats user defined commands specially whose names begin with hook- or hookpost-. It runs hook-foo (hookpost-foo) automatically before (after) a user runs the command foo. In addition, a pseudo-command “stop” exists for when execution stops at a breakpoint.

As an example, consider automatic variable displays. GDB can automatically print the value of expressions every time the program stops with, e.g. display varname. However, what if we want to display all local variables this way?

There’s no direct expression to do it with display, but we can create a hook:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

define hook-stop
    # do it conditionally
    if $display_locals_flag
        # dump the values of all local vars
        info locals
    end
end

# commands to (de)activate the display

define display_locals
    set $display_locals_flag = 1
end

define undisplay_locals
    set $display_locals_flag = 0
end

Python API

Simple helper functions

GDB exposes a Python API for finer control over the debugger. GDB scripts can include Python directly in designated blocks. For instance, right in .gdbinit we can access the Python API to get call stack frame information.

In this example, we’ll trace function calls matching a regex. If no regex is specified, we’ll match all functions visible to GDB, except low level functions (which start with underscore).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

# drop into python to access frame information

python
    # this module contains the GDB API

    import gdb

    # define a helper function we can use later in a user command
    #
    # it prints the name of the function in the specified frame,
    # with indentation depth matching the stack depth

    def frame_indented_name(frame):
        # frame.level() is not always available,
        # so we traverse the list and count depth

        f = frame
        depth = 0
        while (f):
            depth = depth + 1
            f = f.older()
        return "%s%s" % ("  " * depth, frame.name())
end

# trace calls of functions matching a regex

define ftrace
    dont-repeat

    # we'll set possibly many breakpoints, so record the
    # starting number of the group

    set $first_new = 1 + ($bpnum ? $bpnum : 0)

    if $argc < 1
        # by default, trace all functions except those that start with
        # underscore, which are low-level system things
        #
        # rbreak sets multiple breakpoints via a regex

        rbreak ^[a-zA-Z]
    else
        # or match based on ftrace argument, if passed

        rbreak $arg0
    end
    commands
        silent
        
        # drop into python again to use our helper function to
        # print the name of the newest frame

        python print(frame_indented_name(gdb.newest_frame()))

        # then immediately keep going
        cont
    end

    printf "\nTracing enabled. To disable, run:\n\tdel %d-%d\n", $first_new, $bpnum
end

Pretty printing

GDB allows you to customize the way it displays values. For instance, you may want to inspect Unicode strings when working with the ICU library. ICU’s internal encoding for UChar is UTF-16. GDB has no way to know that an array ostensibly containing numbers is actually a string of UTF-16 code units. However, using the Python API, we can convert the string to a form GDB understands.

While a bit esoteric, this example provides the template you would use to create pretty printers for any type.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

import gdb.printing, re

# a pretty printer 

class UCharPrinter:
    'Print ICU UChar string'

    def __init__(self, val):
        self.val = val

    # tell gdb to print the value in quotes, like a string
    def display_hint(self):
        return 'string'

    # the actual work...
    def to_string(self):
        p_c16 = gdb.lookup_type('char16_t').pointer()
        return self.val.cast(p_c16).string('UTF-16')

# bookkeeping that associates the UCharPrinter with the types
# it can handle, and adds an entry to "info pretty-printer"

class UCharPrinterInfo(gdb.printing.PrettyPrinter):
    # friendly name for printer
    def __init__(self):
        super().__init__('UChar string printer')
        self._re = re.compile('^UChar [\[*]')
  
    # is UCharPrinter appropriate for val?
    def __call__(self, val):
        if self._re.match(str(val.type)):
            return UCharPrinter(val)

DDD features

Historical values

Each time the program stops at a breakpoint, DDD records the values of all displayed variables. You can place breakpoints strategically to sample the historical values of a variable, and then view or plot them on a graph.

Interesting shortcuts

DDD has some shortcuts that aren’t obvious from the interface, but which I found interesting in the documentation.

• Control-doubleclick on the left of a line to set a temporary breakpoint, or on an existing breakpoint to delete it. Control double clicking in the data window dereferences in place, rather than creating a new display.

• Click and drag a breakpoint to a new line, and it moves while preserving all its properties.

• Click and hold buttons to reveal special functions. For instance, on the watch button to set a watchpoint on change or on read.

• Pressing Esc (or the interrupt button) acts like an impromptu breakpoint.

• Control-Up/Down changes the stack frame quickly.

• You can display more than single local variables in the data window. Go to Data -> Status Displays to access checkboxes of other common ones, like the backtrace, or all local vars at once.

• Pressing F1 shows help specific to whatever control is under the mouse cursor.

• GDB by default tries to confirm kill/detach when you quit. Use ‘set confirm off’ to disable the prompt.

Further reading

• Debugging with GDB: the GNU Source-Level Debugger. This page links to a printed book, a PDF, and online HTML.
• DDD Manual
• Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems. Timeless debugging techniques, not specific to any particular tooling, or even computers per se.