# Breakpad Processor Library

## Objective

The Breakpad processor library is an open-source framework to access the the
information contained within crash dumps for multiple platforms, and to use that
information to produce stack traces showing the call chain of each thread in a
process. After processing, this data is made available to users of the library.

## Background

The Breakpad processor is intended to sit at the core of a comprehensive
crash-reporting system that does not require debugging information to be
provided to those running applications being monitored. Some existing
crash-reporting systems, such as [GNOME](http://www.gnome.org/)’s Bug-Buddy and
[Apple](http://www.apple.com/)’s
[CrashReporter](http://developer.apple.com/technotes/tn2004/tn2123.html),
require symbolic
information to be present on the end user’s computer; in the case of
CrashReporter, the reports are transmitted only to Apple, not to third-party
developers. Other systems, such as [Microsoft](http://www.microsoft.com/)’s
[Windows Error Reporting](http://msdn.microsoft.com/isv/resources/wer/) and
SupportSoft’s Talkback, transmit only a snapshot of a crashed process’ state,
which can later be combined with symbolic debugging information without the need
for it to be present on end users’ computers. Because symbolic debugging
information consumes a large amount of space and is otherwise not needed during
the normal operation of software, and because some developers are reluctant to
release debugging symbols to their customers, Breakpad follows the latter
approach.

We know of no currently-maintained crash-reporting systems that meet our
requirements, which are to: * allow for symbols to be separate from the
application, * handle crash reports from multiple platforms, * allow developers
to operate their own crash-reporting platform, and to * be open-source. Windows
Error Reporting only functions for Microsoft products, and requires the
involvement of Microsoft’s servers. Talkback, while cross-platform, has not been
maintained and at this point does not support Mac OS X on x86, which we consider
to be a significant platform. Talkback is also closed-source commercial
software, and has very specific requirements for its server platform.

We are aware of Windows-only crash-reporting systems that leverage Microsoft’s
debugging interfaces. Such systems, even if extended to support dumps from other
platforms, are tied to using Windows for at least a portion of the processor
platform.

## Overview

The Breakpad processor itself is written in standard C++ and will work on a
variety of platforms. The dumps it accepts may also have been created on a
variety of systems. The library is able to combine dumps with symbolic debugging
information to create stack traces that include function signatures. The
processor library includes simple command-line tools to examine dumps and
process them, producing stack traces. It also exposes several layers of APIs
enabling crash-reporting systems to be built around the Breakpad processor.

## Detailed Design

### Dump Files

In the processor, the dump data is of primary significance. Dumps typically
contain:

*   CPU context (register data) as it was at the time the crash occurred, and an
    indication of which thread caused the crash. General-purpose registers are
    included, as are special-purpose registers such as the instruction pointer
    (program counter).
*   Information about each thread of execution within a crashed process,
    including:
    *   The memory region used for each thread’s stack.
    *   CPU context for each thread, which for various reasons is not the same
        as the crash context in the case of the crashed thread.
*   A list of loaded code segments (or modules), including:
    *   The name of the file (`.so`, `.exe`, `.dll`, etc.) which provides the
        code.
    *   The boundaries of the memory region in which the code segment is visible
        to the process.
    *   A reference to the debugging information for the code module, when such
        information is available.

Ordinarily, dumps are produced as a result of a crash, but other triggers may be
set to produce dumps at any time a developer deems appropriate. The Breakpad
processor can handle dumps in the minidump format, either generated by an
[Breakpad client “handler”](client_design.md) implementation, or by another
implementation that produces dumps in this format. The
[DbgHelp.dll!MiniDumpWriteDump](http://msdn2.microsoft.com/en-us/library/ms680360.aspx)
function on Windows
produces dumps in this format, and is the basis for the Breakpad handler
implementation on that platform.

The [minidump format](http://msdn.microsoft.com/en-us/library/ms679293%28VS.85%29.aspx) is
essentially a simple container format, organized as a series of streams. Each
stream contains some type of data relevant to the crash. A typical “normal”
minidump contains streams for the thread list, the module list, the CPU context
at the time of the crash, and various bits of additional system information.
Other types of minidump can be generated, such as a full-memory minidump, which
in addition to stack memory contains snapshots of all of a process’ mapped
memory regions.

The minidump format was chosen as Breakpad’s dump format because it has an
established track record on Windows, and it can be adapted to meet the needs of
the other platforms that Breakpad supports. Most other operating systems use
“core” files as their native dump formats, but the capabilities of core files
vary across platforms, and because core files are usually presented in a
platform’s native executable format, there are complications involved in
accessing the data contained therein without the benefit of the header files
that define an executable format’s entire structure. Because minidumps are
leaner than a typical executable format, a redefinition of the format in a
cross-platform header file, `minidump_format.h`, was a straightforward task.
Similarly, the capabilities of the minidump format are understood, and because
it provides an extensible container, any of Breakpad’s needs that could not be
met directly by the standard minidump format could likely be met by extending it
as needed. Finally, using this format means that the dump file is compatible
with native debugging tools at least on Windows. A possible future avenue for
exploration is the conversion of minidumps to core files, to enable this same
benefit on other platforms.

We have already provided an extension to the minidump format that allows it to
carry dumps generated on systems with PowerPC processors. The format already
allows for variable CPUs, so our work in this area was limited to defining a
context structure sufficient to represent the execution state of a PowerPC. We
have also defined an extension that allows minidumps to indicate which thread of
execution requested a dump be produced for non-crash dumps.

Often, the information contained within a dump alone is sufficient to produce a
full stack backtrace for each thread. Certain optimizations that compilers
employ in producing code frustrate this process. Specifically, the “frame
pointer omission” optimization of x86 compilers can make it impossible to
produce useful stack traces given only a stack snapshot and CPU context. In
these cases, however, compiler-emitted debugging information can aid in
producing useful stack traces. The Breakpad processor is able to take advantage
of this debugging information as supplied by Microsoft’s C/C++ compiler, the
only compiler to apply such optimizations by default. As a result, the Breakpad
processor can produce useful stack traces even from code with frame pointer
omission optimizations as produced by this compiler.

### Symbol Files

The [symbol files](symbol_files.md) that the Breakpad processor accepts allow
for frame pointer omission data, but this is only one of their capabilities.
Each symbol file also includes information about the functions, source files,
and source code line numbers for a single module of code. A module is an
individually-loadble chunk of code: these can be executables containing a main
program (`exe` files on Windows) or shared libraries (`.so` files on Linux,
`.dylib` files, frameworks, and bundles on Mac OS X, and `.dll` files on
Windows). Dumps contain information about which of these modules were loaded at
the time the dump was produced, and given this information, the Breakpad
processor attempts to locate debugging symbols for the module through a
user-supplied function embodied in a “symbol supplier.” Breakpad includes a
sample symbol supplier, called `SimpleSymbolSupplier`, that is used by its
command-line tools; this supplier locates symbol files by pathname.
`SimpleSymbolSupplier` is also available to other users of the Breakpad
processor library. This allows for the use of a simple reference implementation,
but preserves flexibility for users who may have more demanding symbol file
storage needs.

Breakpad’s symbol file format is text-based, and was defined to be fairly
human-readable and to encompass the needs of multiple platforms. The Breakpad
processor itself does not operate directly with native symbol formats
([DWARF](http://dwarf.freestandards.org/) and
[STABS](http://sourceware.org/gdb/current/onlinedocs/stabs.html)
on most Unix-like systems,
[.pdb files](http://msdn2.microsoft.com/en-us/library/yd4f8bd1(VS.80).aspx)
on Windows),
because of the complications in accessing potentially complex symbol formats
with slight variations between platforms, stored within different types of
binary formats. In the case of `.pdb` files, the debugging format is not even
documented. Instead, Breakpad’s symbol files are produced on each platform,
using specific debugging APIs where available, to convert native symbols to
Breakpad’s cross-platform format.

### Processing

Most commonly, a developer will enable an application to use Breakpad by
building it with a platform-specific [client “handler”](client_design.md)
library. After building the application, the developer will create symbol files
for Breakpad’s use using the included `dump_syms` or `symupload` tools, or
another suitable tool, and place the symbol files where the processor’s symbol
supplier will be able to locate them.

When a dump file is given to the processor’s `MinidumpProcessor` class, it will
read it using its included minidump reader, contained in the `Minidump` family
of classes. It will collect information about the operating system and CPU that
produced the dump, and determine whether the dump was produced as a result of a
crash or at the direct request of the application itself. It then loops over all
of the threads in a process, attempting to walk the stack associated with each
thread. This process is achieved by the processor’s `Stackwalker` components, of
which there are a slightly different implementations for each CPU type that the
processor is able to handle dumps from. Beginning with a thread’s context, and
possibly using debugging data, the stackwalker produces a list of stack frames,
containing each instruction executed in the chain. These instructions are
matched up with the modules that contributed them to a process, and the
`SymbolSupplier` is invoked to locate a symbol file. The symbol file is given to
a `SourceLineResolver`, which matches the instruction up with a specific
function name, source file, and line number, resulting in a representation of a
stack frame that can easily be used to identify which code was executing.

The results of processing are made available in a `ProcessState` object, which
contains a vector of threads, each containing a vector of stack frames.

For small-scale use of the Breakpad processor, and for testing and debugging,
the `minidump_stackwalk` tool is provided. It invokes the processor and displays
the full results of processing, optionally allowing symbols to be provided to
the processor by a pathname-based symbol supplier, `SimpleSymbolSupplier`.

For lower-level testing and debugging, the processor library also includes a
`minidump_dump` tool, which walks through an entire minidump file and displays
its contents in somewhat readable form.

### Platform Support

The Breakpad processor library is able to process dumps produced on Mac OS X
systems running on x86, x86-64, and PowerPC processors, on Windows and Linux
systems running on x86 or x86-64 processors, and on Android systems running ARM
or x86 processors. The processor library itself is written in standard C++, and
should function properly in most Unix-like environments. It has been tested on
Linux and Mac OS X.

## Future Plans

There are currently no firm plans or timetables to implement any of these
features, although they are possible avenues for future exploration.

The symbol file format can be extended to carry information about the locations
of parameters and local variables as stored in stack frames and registers, and
the processor can use this information to provide enhanced stack traces showing
function arguments and variable values.

On Mac OS X and Linux, we can provide tools to convert files from the minidump
format into the native core format. This will enable developers to open dump
files in a native debugger, just as they are presently able to do with minidumps
on Windows.