# Breakpad Client Libraries

## Objective

The Breakpad client libraries are responsible for monitoring an application for
crashes (exceptions), handling them when they occur by generating a dump, and
providing a means to upload dumps to a crash reporting server. These tasks are
divided between the “handler” (short for “exception handler”) library linked in
to an application being monitored for crashes, and the “sender” library,
intended to be linked in to a separate external program.

## Background

As one of the chief tasks of the client handler is to generate a dump, an
understanding of [dump files](processor_design.md) will aid in understanding the
handler.

## Overview

Breakpad provides client libraries for each of its target platforms. Currently,
these exist for Windows on x86 and Mac OS X on both x86 and PowerPC. A Linux
implementation has been written and is currently under review.

Because the mechanisms for catching exceptions and the methods for obtaining the
information that a dump contains vary between operating systems, each target
operating system requires a completely different handler implementation. Where
multiple CPUs are supported for a single operating system, the handler
implementation will likely also require separate code for each processor type to
extract CPU-specific information. One of the goals of the Breakpad handler is to
provide a prepackaged cross-platform system that masks many of these
system-level differences and quirks from the application developer. Although the
underlying implementations differ, the handler library for each system follows
the same set of principles and exposes a similar interface.

Code that wishes to take advantage of Breakpad should be linked against the
handler library, and should, at an appropriate time, install a Breakpad handler.
For applications, it is generally desirable to install the handler as early in
the start-up process as possible. Developers of library code using Breakpad to
monitor itself may wish to install a Breakpad handler when the library is
loaded, or may only want to install a handler when calls are made in to the
library.

The handler can be triggered to generate a dump either by catching an exception
or at the request of the application itself. The latter case may be useful in
debugging assertions or other conditions where developers want to know how a
program got in to a specific non-crash state. After generating a dump, the
handler calls a user-specified callback function. The callback function may
collect additional data about the program’s state, quit the program, launch a
crash reporter application, or perform other tasks. Allowing for this
functionality to be dictated by a callback function preserves flexibility.

The sender library is also has a separate implementation for each supported
platform, because of the varying interfaces for accessing network resources on
different operating systems. The sender transmits a dump along with other
application-defined information to a crash report server via HTTP. Because dumps
may contain sensitive data, the sender allows for the use of HTTPS.

The canonical example of the entire client system would be for a monitored
application to link against the handler library, install a Breakpad handler from
its main function, and provide a callback to launch a small crash reporter
program. The crash reporter program would be linked against the sender library,
and would send the crash dump when launched. A separate process is recommended
for this function because of the unreliability inherent in doing any significant
amount of work from a crashed process.

## Detailed Design

### Exception Handler Installation

The mechanisms for installing an exception handler vary between operating
systems. On Windows, it’s a relatively simple matter of making one call to
register a [top-level exception
filter](http://msdn.microsoft.com/library/en-us/debug/base/setunhandledexceptionfilter.asp)
callback function. On most Unix-like systems such as Linux, processes are
informed of exceptions by the delivery of a signal, so an exception handler
takes the form of a signal handler. The native mechanism to catch exceptions on
Mac OS X requires a large amount of code to set up a Mach port, identify it as
the exception port, and assign a thread to listen for an exception on that port.
Just as the preparation of exception handlers differ, the manner in which they
are called differs as well. On Windows and most Unix-like systems, the handler
is called on the thread that caused the exception. On Mac OS X, the thread
listening to the exception port is notified that an exception has occurred. The
different implementations of the Breakpad handler libraries perform these tasks
in the appropriate ways on each platform, while exposing a similar interface on
each.

A Breakpad handler is embodied in an `ExceptionHandler` object. Because it’s a
C++ object, `ExceptionHandler`s may be created as local variables, allowing them
to be installed and removed as functions are called and return. This provides
one possible way for a developer to monitor only a portion of an application for
crashes.

### Exception Basics

Once an application encounters an exception, it is in an indeterminate and
possibly hazardous state. Consequently, any code that runs after an exception
occurs must take extreme care to avoid performing operations that might fail,
hang, or cause additional exceptions. This task is not at all straightforward,
and the Breakpad handler library seeks to do it properly, accounting for all of
the minute details while allowing other application developers, even those with
little systems programming experience, to reap the benefits. All of the Breakpad
handler code that executes after an exception occurs has been written according
to the following guidelines for safety at exception time:

*   Use of the application heap is forbidden. The heap may be corrupt or
    otherwise unusable, and allocators may not function.
*   Resource allocation must be severely limited. The handler may create a new
    file to contain the dump, and it may attempt to launch a process to continue
    handling the crash.
*   Execution on the thread that caused the exception is significantly limited.
    The only code permitted to execute on this thread is the code necessary to
    transition handling to a dedicated preallocated handler thread, and the code
    to return from the exception handler.
*   Handlers shouldn’t handle crashes by attempting to walk stacks themselves,
    as stacks may be in inconsistent states. Dump generation should be performed
    by interfacing with the operating system’s memory manager and code module
    manager.
*   Library code, including runtime library code, must be avoided unless it
    provably meets the above guidelines. For example, this means that the STL
    string class may not be used, because it performs operations that attempt to
    allocate and use heap memory. It also means that many C runtime functions
    must be avoided, particularly on Windows, because of heap operations that
    they may perform.

A dedicated handler thread is used to preserve the state of the exception thread
when an exception occurs: during dump generation, it is difficult if not
impossible for a thread to accurately capture its own state. Performing all
exception-handling functions on a separate thread is also critical when handling
stack-limit-exceeded exceptions. It would be hazardous to run out of stack space
while attempting to handle an exception. Because of the rule against allocating
resources at exception time, the Breakpad handler library creates its handler
thread when it installs its exception handler. On Mac OS X, this handler thread
is created during the normal setup of the exception handler, and the handler
thread will be signaled directly in the event of an exception. On Windows and
Linux, the handler thread is signaled by a small amount of code that executes on
the exception thread. Because the code that executes on the exception thread in
this case is small and safe, this does not pose a problem. Even when an
exception is caused by exceeding stack size limits, this code is sufficiently
compact to execute entirely within the stack’s guard page without causing an
exception.

The handler thread may also be triggered directly by a user call, even when no
exception occurs, to allow dumps to be generated at any point deemed
interesting.

### Filter Callback

When the handler thread begins handling an exception, it calls an optional
user-defined filter callback function, which is responsible for judging whether
Breakpad’s handler should continue handling the exception or not. This mechanism
is provided for the benefit of library or plug-in code, whose developers may not
be interested in reports of crashes that occur outside of their modules but
within processes hosting their code. If the filter callback indicates that it is
not interested in the exception, the Breakpad handler arranges for it to be
delivered to any previously-installed handler.

### Dump Generation

Assuming that the filter callback approves (or does not exist), the handler
writes a dump in a directory specified by the application developer when the
handler was installed, using a previously generated unique identifier to avoid
name collisions. The mechanics of dump generation also vary between platforms,
but in general, the process involves enumerating each thread of execution, and
capturing its state, including processor context and the active portion of its
stack area. The dump also includes a list of the code modules loaded in to the
application, and an indicator of which thread generated the exception or
requested the dump. In order to avoid allocating memory during this process, the
dump is written in place on disk.

### Post-Dump Behavior

Upon completion of writing the dump, a second callback function is called. This
callback may be used to launch a separate crash reporting program or to collect
additional data from the application. The callback may also be used to influence
whether Breakpad will treat the exception as handled or unhandled. Even after a
dump is successfully generated, Breakpad can be made to behave as though it
didn’t actually handle an exception. This function may be useful for developers
who want to test their applications with Breakpad enabled but still retain the
ability to use traditional debugging techniques. It also allows a
Breakpad-enabled application to coexist with a platform’s native crash reporting
system, such as Mac OS X’ [CrashReporter](http://developer.apple.com/technotes/tn2004/tn2123.html)
and [Windows Error Reporting](http://msdn.microsoft.com/isv/resources/wer/).

Typically, when Breakpad handles an exception fully and no debuggers are
involved, the crashed process will terminate.

Authors of both callback functions that execute within a Breakpad handler are
cautioned that their code will be run at exception time, and that as a result,
they should observe the same programming practices that the Breakpad handler
itself adheres to. Notably, if a callback is to be used to collect additional
data from an application, it should take care to read only “safe” data. This
might involve accessing only static memory locations that are updated
periodically during the course of normal program execution.

### Sender Library

The Breakpad sender library provides a single function to send a crash report to
a crash server. It accepts a crash server’s URL, a map of key-value parameters
that will accompany the dump, and the path to a dump file itself. Each of the
key-value parameters and the dump file are sent as distinct parts of a multipart
HTTP POST request to the specified URL using the platform’s native HTTP
facilities. On Linux, [libcurl](http://curl.haxx.se/) is used for this function,
as it is the closest thing to a standard HTTP library available on that
platform.

## Future Plans

Although we’ve had great success with in-process dump generation by following
our guidelines for safe code at exception time, we are exploring options for
allowing dumps to be generated in a separate process, to further enhance the
handler library’s robustness.

On Windows, we intend to offer tools to make it easier for Breakpad’s settings
to be managed by the native group policy management system.

We also plan to offer tools that many developers would find desirable in the
context of handling crashes, such as a mechanism to determine at launch if the
program last terminated in a crash, and a way to calculate “crashiness” in terms
of crashes over time or the number of application launches between crashes.

We are also investigating methods to capture crashes that occur early in an
application’s launch sequence, including crashes that occur before a program’s
main function begins executing.