documents/DEVELOPMENT.md

# Android Rust Toolchain Development Tasks

Development of the Android Rust toolchain can involve a large number of tools
and technologies, many of which are open source and have better documentation
elsewhere.  This file will focus on describing the tools developed as part of
this project.  For a definitive list of the options supported by each tool
please use the `--help` flag.

* [Audit prebuilts for shared library usage](#audit)
* [Benchmark the performance of one or more prebuilts](#benchmark)
* [BOLT optimize a prebuilt](#boltifyer)
* [Build a Rust toolchain prebuilt](#build)
* [Fetch new Rust toolchain source](#fetch-source)
* [Ensure LTO is performed on the toolchain](#lto-check)
* [Merge PGO and BOLT profiles](#merge-profiles)
* [Run the build pipeline](#pipeline)
* [Recontextualize patches](#recontextualize-patches)
* [Analyze soong traces](#soong-trace)
* [Test the compiler](#test-compiler)
* [Update Android Rust toolchain prebuilts](#update-prebuilts)
* [Running tools inside Docker](#running-tools-in-docker)

## Audit

The [`toools/audit.py`](../tools/audit.py) script can be used to check a
prebuilt archive for any new shared library dependencies.  New dependencies
usually indicate a hermeticity issue and should be investigated to ensure that
the new library originated from in-tree source or resolves to a prebuilt
distributed as part of Android.

Usage:

```shell
./tools/audit.py <scandir>
```

To see a mapping between binaries and their dependencies you can pass the
`--map` flag.

If a new shared library is required by the toolchain its name should be added
to the `resources/shared_library_allow_list.txt` file.  This will enable
`audit.py` to recognize the library and not fail the audit when it's present as
a dependency.

## Benchmark

One or more toolchain prebuilts can be benchmarked using the
[`tools/benchmark.py`](../tools/benchmark.py) script.  This script will build
Android with each of the prebuilts some number of times (default of 5) and
produce a CSV file containing timing values for various categories of targets
such as C/C++, Java or Rust.

An example invocation of the benchmarking tool:
```shell
$ ./tools/benchmark.py --label example-invoke --iterations 10 ../../dist/rust-baseline.tar.xz ../../dist/rust-experiment.tar.xz
```

## BOLTifyer

The `bolt` command line tool can be a bit finnicky to work with and so we use
the [`tools/boltifyer.py`](../tools/boltifyer.py) script to automate optimizing
an entire prebuilt at a time.  By default the tool will apply peephole
optimizations to all target binaries in the archive.  The `--profile-generate`
and `--profile-use` options can be set generate or use profiles in a fashion
similar to how normal PGO profiles are handled.  These flags take absolute paths
as arguments.  If you'd like the profiles to be generated into a path relative
to the instrumented binary you can use the `--profile-generate-relative` flag.

Most BOLT optimizations require relocation information to be useful.  Make sure
to pass the `--emit-relocs` flag to the `tools/build.py` script if you intend
to BOLT the prebuilt later.

Example usage:
```shell
$ ./tools/boltifyer.py --bulid-name 1.77.1-bolted --profile-generate -- ../../dist/rust-1.77.1-relocs.tar.xz
```

As no value was passed to the `--profile-generate` flag in the above command
the tool will use the default location: `out/profiles`.

## Build

Juggling the configuration values for each of the targets we support is too
burdensome and error prone to do by hand.  As such, the Android Rust toolchain
uses the [`tools/build.py`](../tools/build.py) script to set-up a new copy of
the Rust source in the `out/` directory, apply patches, emit compiler wrapper
scripts, initialize environment variables, and invoke the Rust build system
(`x.py`).

For a comprehensive list of the options available please see the output from
the `--help` flag.  Below are several example invocations of the script that
can be used for common tasks.

Linux release:
```shell
$ ./tools/build.py --lto thin --llvm-linkage shared --cgu1
```

Fast testing of host targets:
```shell
$ ./tools/build.py --lto thin --llvm-linkage shared --host-only
```

You an re-use existing intermediate outputs by telling the script to avoid
re-initializing the Rust out directory:
```shell
$ ./tools/build.py --lto thin --llvm-linkage shared --host-only --no-copy-and-patch
```

The flags controlling profiling instrumentation for the compiler take paths to
where the profiles should be stored but will default to a value of
`/path/to/android/out/profiles` if the flag is set but no path is given.  The
same default paths are used by `tools/test_compiler.py`.

The following artifacts will be placed into the distribution (`--dist`)
directory:

* Archive containing the Rust toolchain
* Android manifest (with commit SHAs) used to build the toolchain
* Build command used to build toolchain
* PGO and BOLT profiles used to optimize the toolchain

## Fetch Source

Upstream Rust source archives can be downloaded, extracted, and committed to
Android using the [`tools/fetch_source.py`](../tools/fetch_source.py) script.
By default the tool will fetch the release archive for the provided version
number if it exists.  To fetch the beta or nightly versions of the Rust source
use the `--beta` and `--nightly` flags respectively.  Branch names and issue
numbers can be provided using the `--branch` and `--issue` flags.

```shell
$ ./tools/fetch_source.py --issue 333887339 1.78.0
```

## LTO Check

The [`tools/lto_check.py`](../tools/lto_check.py) script takes no argument and
will simply check all `.o` objects under `out/rustc/build` and list any that do
not contain LLVM IR.

## Merge Profiles

To get the best performance out of Android's Rust toolchain we want to optimize
it using profiles generated while it compiles for each of the backends we
utilize for production code (ARM, RISC-V, and x86).  While there exists tools
for merging multiple PGO or BOLT profiles into a single, representative file
(`profdata` and `merge_fdata` respectively), these tools can require multiple
invocations and are generally a pain to use manually for a project this large.

The [`./tools/merge_profiles.py`](../tools/merge_profiles.py) script takes the
paths to multiple profile locations and produces a single profile for PGO or an
archive of profiles for BOLT.

```shell
$ ./tools/merge_profiles.py ../../dist/run1 ../../dist/run2
```

## Pipeline

The steps necessary to build a profiled and optimized version of the Rust
toolchain are numerous and finnicky.  To help avoid mistakes and enable rapid
testing and debugging the [`tools/pipeline.py`](../tools/pipeline.py) script
will execute all of the build stages locally.  The `--llvm-linkage` and `--lto`
flags are shared with the `./tools/build.py` script but all other options are
determined by the stages in the pipeline.  Artifacts generated by the pipeline
script are placed into a `build-name` specific subdirectory of the distribution
directory.  For example, if `1.78.0-pipeline` is provided as the `build-name`
artifacts will be found under `/path/to/dist/rust-1.78.0-pipeline/stage{0-6}/`.

The stage at which the script starts, resumes, or stops compilation can be
controlled with the `--start` and `--end` flags.  The `--overwrite` flag needs
to be provided if you use the same `build-name` and wish to re-run a stage that
has already completed successfully.

For a full description of the optimization pipeline please see the
[BUILDS](./BUILDS.md#pgo-pipeline) file.


## Recontextualize Patches

Android maintains several patches against the upstream Rust toolchain source.
These patches range from small changes that are needed locally but can't be
upstreamed, patches that have been backported or already upstreamed, or patches
that are under development and are not ready for external release.  The
[`tools/recontextualize_patches.py`](../tools/recontextualize_patches.py) script
helps keep these patches up-to-date relative to the upstream source.  When this
tool is run it will re-apply the patches to the `toolchain/rustc` source
repository using the `git am` command before re-emitting them into the
`toolchain/android_rust/patches` directory.  If the patches apply successfully
the result will be refreshed context information.  By performing this step as
part of every toolchain release we can avoid context drift and the eventual
breaking of patches that that entails.

## Soong Trace

The [`tools/soong_trace.py`](../tools/soong_trace.py) script is used to extract
compiler statistics from a Soong trace file.  It will print the number of
targets, and time spent compiling them, for C/C++, Java, and Rust.  The `--csv`
option can be used to cause to tool to generate a CSV file containing the same
data.

## Test Compiler

Being able to compile a toolchain is of little use to us if it can't, in turn,
compile Android.  To help ensure that we ship a working toolchain we use the
[`tools/test_compiler.py`](../tools/test_compiler.py) script to verify the state
of our prebuilts.

The `--clean` flag can be used to ensure that the `out/` directory is fully
deleted before we attempt to build any Android targets.  The following flags
can be used to select which Android targets to build:

* `--all-rust` - Every Rust target defined for the `lunch`ed device
* `--image` - All targets necessary to generate a boot image
* `--test-targets` - Android targets needed to run device tests
* `--rustdoc` - All Rustdoc targets
* `--custom-targets` - A set of user-specified Android targets

Note: not all Rust targets build for the RISC-V architecture.  It is best to
restrict the test targets to `--image` until architecture support has matured.

This tool is also capable of collecting profiles from the test compiler.  Use
the following (mutually exclusive) flags to collect the relevant profiles:

* `--profile-generate`
* `--cs-profile-generate`
* `--bolt-profile-generate`

Basic usage:
```shell
$ ./tools/test_compiler.py --prebuilt-path ../../dist/rust-1.77.1.tar.xz --clean --all-rust
```

Re-use the compiler in the test directory to build custom targets:
```shell
$ ./tools/test_compiler.py --reuse-prebuilt --clean --custom-targets keystore2 libtokio
```

## Update Prebuilts

Updating the Rust prebuilts used by Android involves commits to three separate
projects:

* `build/soong` - Rust version number
* `prebuilts/rust` - Rust executables and libraries
* `toolchain/android_rust` - Build command, profiles, and manifest used to build release

The [`tools/update_prebuilts.py`](../tools/update_prebuilts.py) script will
create the three Git commits but they will need to be uploaded manually.

Basic usage of the tools only requires the Rust version number (though an issue
number is also strongly recommended):

```shell
$ ./tools/update_prebuilts.py --issue 333887339 1.78.0
```

The `--bid` flag can be used to select a specific build, which can be helpful
if "last known good build" detection feature encounters an error.  The
`--bolt`/`--no-bolt`, `--musl`/`--no-musl`, and `--pgo`/`--no-pgo` flags can be
used to select which build targets to download artifacts from.

## Running Tools in Docker

The Android Rust toolchain provides a Docker image that can be used when
building the toolchain and running other tools.  This Docker image is meant to
be used on the Android build servers to provide a stable and reproducible build
environment.

To use this Docker image you'll first need to build it. To do so enter the
`toolchain/android_rust/resources` directory and run the following command:

```shell
$ docker build -t android-rust:dev .
```

You may then use the [`tools/docker_run.py`](../tools/docker_run.py) script to
execute a tool inside an instance of the docker image like so:

```shell
$ ./tools/docker_run.py ./tools/build.py --build-name docker-test
```

To run start an interactive Docker instance with the correct mount point for the
Android source you can run this command (substituting in the correct path to
your Android root):

```shell
$ sudo docker run --mount type=bind,source=/path/to/android/root,target=/android -it android-rust:dev /bin/bash
```

### Updating the Docker Image

The Android Rust Toolchain Docker image is defined in
[resources/Dockerfile](../resources/Dockerfile) and modifications can be built
and published to the gCloud artifact registry by a team member.

To update the Docker image you first need to configure your gCloud project and
obtain credentials:

```shell
$ gcloud config set project android-native-docker
$ gcloud auth login
```

Next, you can run the following command in the `resources/` directory:

```shell
$ gcloud builds submit --region=us-west1 --tag us-west1-docker.pkg.dev/android-native-docker/rust/toolchain-build-image:v1
```