documents/BUILDS.md

# Android Rust Toolchain Branches and Builds

The basic building blocks of Android's continuous integration (CI)
infrastructure are the _branch_, _build target_, and _build_.  These are
distinct from the Git, Soong, and `lunch` concepts of the same name and when
these terms are used in this document they should be interpreted in the context
of the Android CI infrastructure.

Each branch has one or more build targets and these targets will usually (see
[ABTD](#android-build-and-test-on-demand)) be executed as part of a batch of
jobs in response to a build trigger.  Both the individual job and the batch
are, confusingly, referred to as _builds_ in Android documenation and parlance.
In this document we'll refer to the batch of builds as a _build group_.  Each
build in a build group will all share the same _build ID_.

The scheduling of a build group might be caused by one of several factors:

* `cron`: Trigger builds at defined intervals
* Post-submit: Builds triggered by commits to specific projects
* Pre-submit: Builds triggered by Tree Hugger/Gerrit to test CLs that touch specific files

Build groups triggered by `cron` are a subset of the post-submit builds.  Some
post-submit builds are paired with tests that will either execute on a buildbot
(host) or a device.

All post-submit builds can be monitored using the
[Build Explorer](https://android-build.corp.google.com/build_explorer/).  When
a pre-submit build is triggered by Tree Hugger the exit status of the job will
be compared to a reference post-submit build.  If the reference build exited
successfully (green) and the application of the CL results in a broken (red)
build or failed tests Tree Hugger reports a failure.

## Build Chaining

In Android, one build depending on the outputs of another build is referred to
as a _build chain_.  The Android Rust toolchain makes use of build chaining to

1. Instrument, profile, and optimize the toolchain
2. Obtain released NDKs
3. Test pre-release NDKs
4. Re-use compilation artifacts to reduce work done by buildbots

When a build target depends on the output of another target in the same branch
the job for the dependent target will be scheduled after the dependencies are
satisfied.  If a build target depends on a target in a _different_ branch it
will be built twice.  The first time will be in response to the original build
trigger and the artifact dependencies will be pulled form the
_last known good build_ of the depended upon target.  A second build of the
target will be triggered when the new build of the depended upon target
(launched by the same trigger) is completed, and will use the newly produced
artifacts.

The cross-branch chaining behavior can sometimes result in the first of the two
builds failing.  Specifically, this can happen in the
[`rustc-linux_pgo_bolt`](./BUILDS.md#rust-linux_pgo_bolt) build when the BOLT
profiles for an older version of the Rust toolchain are provided to a build of
newer Rust toolchain source.  Check the build log for error messages related to
missing or incorrect artifact inputs and then wait for the second build to
complete.

## Scheduling Pool

When a build is created it must be matched to a buildbot to begin executing.
To do this, Launch Control will iterate over the _scheduling pool_
[definitions](http://google3/wireless/android/launchcontrol/config/prod/scheduling_pool_config.asciipb),
in order, and find the first pool that matches the branch and target names of
the new build.

There are several important concepts to keep in mind when reasoning about
and/or modifying the scheduling pools:

1. An individual buildbot may be a member of more than one scheduling pool.

2. _Exclusive_ pools prevent the scheduling algorithm from allocating the
pinned targets to any pool besides the one marked `exclusive: true`.

3. [Trident](https://g3doc.corp.google.com/wireless/android/build_tools/g3doc/internal/trident.md)
enabled builds need to be mapped to specific Trident scheduling pools that we
do not have administrative rights to.  Trident is only required for builds that
are defined in a Droid Monitored branch *and* enabled in pre-submit.  As none
of our branches are Droid Monitored and we want better control over our build
scheduling none of our builds should be Trident enabled.

4. Each scheduling pool only supports a single `build_type`.  As such, separate
pools need to be defined for pre- and post-submit builds.

5. Incremental builds are very finnicky.  If any job that is not the same build
target from the same branch is mapped to a build bot that previously hosted an
incremental build the new build will delete the incremental state.  This means
that each incremental job must be mapped to a non-shared and exclusive pool
whose machines are not members of other pools.

6. TODO: Add information about default pools when that is sorted out.

TODO: If we are still going to use a custom scheduling pool a description of
the pools and the logic behind their allocation should go here.

## Branches

Due to https://source.android.com/docs/whatsnew/site-updates#aosp-changes,
the Rust toolchain branch is now developed internally in Android. Luckily, we
also provide a mirror of our sources using the
`mirror-goog-main-rust-toolchain-source` branch. In the future, we will provide
instructions for how to best reproduce our mirrored top-of-tree builds in
public.

### [`main-rust-toolchain`](https://android-build.corp.google.com/build_explorer/branch/git_main-rust-toolchain)

The `main-rust-toolchain` branch is used to generate artifacts for release and
thus has a minimal set of post-submit triggers, pre-submit triggers only on the
manifest, and no `cron` triggers.

* [Branch definition](http://google3/configs/wireless/android/busytown/platform/toolchain/git_main-rust-toolchain.gcl)
* [Pre-submit builds](http://google3/configs/wireless/android/busytown/presubmit/prod/branch/git_main-rust-toolchain.gcl)

#### `rust-darwin_mac`

A Darwin Rust toolchain

* Host: `darwin-x86`
* Script: `tools/build.py`
* Arguments: `--llvm-linkage shared --cgu1`

#### `rust-linux_glibc`

A Linux Rust toolchain built with glibc

* Host: `linux-x86`
* Script: `tools/build.py`
* Arguments: `--llvm-linkage shared --lto thin --cgu1`

#### `rust-linux_musl`

A Linux Rust toolchain built with MUSL

* Host: `linux-x86`
* Script: `tools/build.py`
* Arguments: `--llvm-linkage shared`

#### `rust-linux_pgo_bolt`

A Linux Rust toolchain built with glibc and optimized using PGO and BOLT
profiles from the [profiling pipeline](#stage5-merge)

* Host: `linux-x86`
* Script: `tools/build.py`
* Arguments: `--llvm-linkage shared --lto thin --cgu1 --profile-use <path-to-profiles> --bolt-profile-use <path-to-profiles>`

#### `rust-windows_gnu`

A cross-compilation build of the Windows toolchain on a Linux host

* Host: `linux-x86`
* Script: `tools/build.py`
* Arguments: `--llvm-linkage static`

#### `rust-windows_gnu_native`

The native build of the Windows host toolchain

* Host: `windows-x86`
* Script: `tools/build.py`
* Arguments: `--llvm-linkage static --cgu1`

### [`main-plus-rust`](https://android-build.corp.google.com/build_explorer/branch/git_main-plus-rust)

* [Branch definition](http://google3/configs/wireless/android/busytown/platform/toolchain/git_main-plus-rust.gcl)
* [Pre-submit builds](http://google3/configs/wireless/android/busytown/presubmit/prod/branch/git_main-plus-rust.gcl)
* [Pre-submit tests](http://google3/configs/wireless/android/busytown/platform/toolchain/git_main-rust-profiling.gcl)

#### `cf_arm64_phone-trunk-userdebug`

Build Android's Rust code with the compiler produced by `rust-baseline`.  This
target also produces an Android image used for running emulated device tests.

* Host: `linux-x86`
* Script: `tools/test_compiler.py`
* Arguments: `--all-rust --image --target aosp_cf_arm64_phone --variant userdebug --release trunk`

#### `cf_riscv64_phone-trunk-userdebug`

Build Android's Rust code with the compiler produced by `rust-baseline`.

* Host: `linux-x86`
* Script: `tools/test_compiler.py`
* Arguments: `--image --target aosp_cf_riscv64_phone --variant userdebug --release trunk`

#### `cf_x86_64_phone-trunk-userdebug`

Build Android's Rust code with the compiler produced by `rust-baseline`.  This
target also produces an Android image used for running emulated device tests.

* Host: `linux-x86`
* Script: `tools/test_compiler.py`
* Arguments: `--all-rust --image --target aosp_cf_x86_64_phone --variant userdebug --release trunk`

#### `cf_x86_64_phone-trunk-user`

Build Android's Rust code with the compiler produced by `rust-baseline`.  This
target tests the `user` build variant.

* Host: `linux-x86`
* Script: `tools/test_compiler.py`
* Arguments: `--all-rust --image --target aosp_cf_x86_64_phone --variant user --release trunk`

#### `cf_x86_64_phone-trunk-eng`

Build Android's Rust code with the compiler produced by `rust-baseline`.  This
target tests the `eng` build variant.

* Host: `linux-x86`
* Script: `tools/test_compiler.py`
* Arguments: `--all-rust --image --target aosp_cf_x86_64_phone --variant eng --release trunk`

#### `pixel-trunk-userdebug`

Build Android's Rust code with the compiler produced by `rust-baseline`.  This
target also produces an Android image used for running device tests on the most
recent in-market Pixel device.

* Host: `linux-x86`
* Script: `tools/test_compiler.py`
* Arguments: `--all-rust --image --target aosp_husky --variant eng --release trunk`

#### `rustdoc`

Build all of the Rust documentation in Android.

* Host: `linux-x86`
* Script: `tools/test_compiler.py`
* Arguments: `--rustdoc --target aosp_cf_x86_64_phone --variant userdebug --release trunk`

#### `boltifyer-instrument`

Use checked-in resources to add BOLT instrumentation to a Rust toolchain.  This
helps to verify that our BOLT tooling is working without having to build a
fresh toolchain.

* Host: `linux-x86`
* Script: `tools/boltifyer.py`
* Arguments: `--profile-generate -- resources/rust-1.78.0-relocs.tar.xz`

#### `boltifyer-optimize`

Use checked-in resources to optimize a Rust toolchain using BOLT.  This
helps to verify that our BOLT tooling is working without having to build a
toolchain, instrument it, and then profile the compiler.

* Host: `linux-x86`
* Script: `tools/boltifyer.py`
* Arguments: `--profile-use resources/bolt-profiles-1.78.0.tar.xz resources/rust-1.78.0-relocs.tar.xz`

#### `merge_profiles-bolt`

Use checked-in resources to ensure we can successfully merge BOLT profiles.

* Host: `linux-x86`
* Script: `tools/merge_profiles.py`
* Arguments: `resources/profiles/cf_arm64 resources profiles/cf_riscv64 resources/profiles/cf_x86_64`

#### `merge_profiles-pgo`

Use checked-in resources to ensure we can successfully merge PGO profiles.

* Host: `linux-x86`
* Script: `tools/merge_profiles.py`
* Arguments: `resources/profiles/cf_arm64/bolt-profiles.tar.xz resources profiles/cf_riscv64/bolt-profiles.tar.xz resources/profiles/cf_x86_64/bolt-profiles.tar.xz`

#### `rust-baseline`

This target builds a Linux Rust toolchain as a method of testing the
toolchain's build system.  The resulting compiler is then used by other build
targets in this branch to ensure that it produces correct output.

* Host: `linux-x86`
* Script: `tools/build.py`
* Arguments: `--llvm-linkage shared --lto thin`

#### `rust-linux_ndk`

A canary build for the NDK top-of-tree builds.  This build is meant to alert us
to problems in upcoming NDK releases before they occur.

* Host: `linux-x86`
* Script: `tools/build.py`
* Arguments: `--llvm-linkage shared --lto thin --no-bare-targets --no-host-multilibs`

#### `test_suites_x86_64`

This build produces artifacts for running host and device tests.  The build
target definition is a copy of the target of the same name in `main`.

* Host: `linux-x86`
* Script: `<android root>/build/soong/soong_ui.bash`
* Arguments: `--make-mode cts dist DIST_DIR=%dist_dir% TARGET_PRODUCT=aosp_x86_64 TARGET_RELEASE=trunk_staging WITH_DEXPREOPT_BOOT_IMG_AND_SYSTEM_SERVER_ONLY=true acts_tests vts tradefed-all general-tests test_mapping robolectric-tests host-unit-tests`

### [`main-rust-profiling`](https://android-build.corp.google.com/build_explorer/branch/git_main-rust-profiling)

This branch is responsible for generating PGO and BOLT profiles for use by the
release branch.  A build is scheduled once a week on the weekends via `cron` as
well as whenever there are commits to `toolchain/rustc`.  If you need to
manually trigger the creation of new profiles you can create an empty commit to
the `toolchain/rustc` project.

The build targets in each stage depend on the output of the previous stage,
with the `stage2-merge` and `stage5-merge` build targets acting as merge points
for the stages with multiple targets.  A full run of this pipeline can take
over six hours to complete.

* [Branch definition](http://google3/configs/wireless/android/busytown/platform/toolchain/git_main-rust-profiling.gcl)
* [Pre-submit builds](http://google3/configs/wireless/android/busytown/presubmit/prod/branch/git_main-rust-profiling.gcl)

#### `stage1-rust-pgo-inst`

Build a Linux Rust toolchain with PGO instrumentation.

* Host: `linux-x86`
* Script: `tools/build.py`
* Arguments: `--llvm-linkage shared --lto thin --cgu1 --profile-generate`

#### `stage2-cf_arm64_phone-trunk-userdebug`

Use the toolchain from `stage1-rust-pgo-inst` to generate PGO profiles while
compiling Android Rust targets for `cf_arm64`.

* Host: `linux-x86`
* Script: `tools/test_compiler.py`
* Arguments: `--all-rust --profile-generate --target aosp_cf_arm64_phone --variant userdebug --release trunk`

#### `stage2-cf_riscv64_phone-trunk-userdebug`

Use the toolchain from `stage1-rust-pgo-inst` to generate PGO profiles while
compiling Android Rust targets for `cf_riscv64`.

* Host: `linux-x86`
* Script: `tools/test_compiler.py`
* Arguments: `--image --profile-generate --target aosp_cf_riscv64_phone --variant userdebug --release trunk`

#### `stage2-cf_x86_64_phone-trunk-userdebug`

Use the toolchain from `stage1-rust-pgo-inst` to generate PGO profiles while
compiling Android Rust targets for `cf_x86_64`.

* Host: `linux-x86`
* Script: `tools/test_compiler.py`
* Arguments: `--all-rust --profile-generate --target aosp_cf_x86_64_phone --variant userdebug --release trunk`

#### `stage2-merge`

Merge profiles from the `stage2-cf_*` targets into a single profile.

* Host: `linux-x86`
* Script: `tools/merge_profiles.py`
* Arguments: `<list of paths to profiles>`

#### `stage3-rust-pgo-opt`

Build a Linux Rust toolchain using the profile from `stage2-merge` to preform
profile-guided optimization.  This toolchain contains the relocation
information required by BOLT in later stages.

* Host: `linux-x86`
* Script: `tools/build.py`
* Arguments: `--llvm-linkage shared --lto thin --cgu1 --profile-use --emit-relocs`

#### `stage4-rust-bolt-inst`

Add BOLT instrumentation to the Rust toolchain generated by
`stage3-rust-pgo-opt`.

* Host: `linux-x86`
* Script: `tools/boltifyer.py`
* Arguments: `--profile-generate`

#### `stage5-cf_arm64_phone-trunk-userdebug`

Use the toolchain from `stage4-rust-bolt-inst` to generate BOLT profiles while
compiling Android Rust targets for `cf_arm64`.

* Host: `linux-x86`
* Script: `tools/test_compiler.py`
* Arguments: `--all-rust --bolt-profile-generate --target aosp_cf_arm64_phone --variant userdebug --release trunk`

#### `stage5-cf_riscv64_phone-trunk-userdebug`

Use the toolchain from `stage4-rust-bolt-inst` to generate BOLT profiles while
compiling Android Rust targets for `cf_riscv64`.

* Host: `linux-x86`
* Script: `tools/test_compiler.py`
* Arguments: `--image --bolt-profile-generate --target aosp_cf_riscv64_phone --variant userdebug --release trunk`

#### `stage5-cf_x86_64_phone-trunk-userdebug`

Use the toolchain from `stage4-rust-bolt-inst` to generate BOLT profiles while
compiling Android Rust targets for `cf_x86_64`.

* Host: `linux-x86`
* Script: `tools/test_compiler.py`
* Arguments: `--all-rust --bolt-profile-generate --target aosp_cf_x86_64_phone --variant userdebug --release trunk`

#### `stage5-merge`

Merge profiles from the `stage5-cf_*` targets into a single profile.

* Host: `linux-x86`
* Script: `tools/merge_profiles.py`
* Arguments: `<list of paths to profiles>`

#### `stage6-rust-bolt-opt`

Use the BOLT profiles from `stage5-merge` to optimize the Rust toolchain
generated by `stage3-rust-pgo-opt`.

* Host: `linux-x86`
* Script: `tools/boltifyer.py`
* Arguments: `--profile-use`

#### `stage7-cf_arm64_phone-trunk-userdebug`

Verify that the compiler produced by `stage6-rust-bolt-opt` can compile the
Android Rust codebase.

* Host: `linux-x86`
* Script: `tools/test_compiler.py`
* Arguments: `--all-rust --target aosp_cf_arm64_phone --variant userdebug --release trunk`