Android Rust Toolchain Branches and Builds

The basic building blocks of Android's continuous integration (CI) infrastructure are the branch, build target, and build. These are distinct from the Git, Soong, and lunch concepts of the same name and when these terms are used in this document they should be interpreted in the context of the Android CI infrastructure.

Each branch has one or more build targets and these targets will usually (see ABTD) be executed as part of a batch of jobs in response to a build trigger. Both the individual job and the batch are, confusingly, referred to as builds in Android documenation and parlance. In this document we'll refer to the batch of builds as a build group. Each build in a build group will all share the same build ID.

The scheduling of a build group might be caused by one of several factors:

  • cron: Trigger builds at defined intervals
  • Post-submit: Builds triggered by commits to specific projects
  • Pre-submit: Builds triggered by Tree Hugger/Gerrit to test CLs that touch specific files

Build groups triggered by cron are a subset of the post-submit builds. Some post-submit builds are paired with tests that will either execute on a buildbot (host) or a device.

All post-submit builds can be monitored using the Build Explorer. When a pre-submit build is triggered by Tree Hugger the exit status of the job will be compared to a reference post-submit build. If the reference build exited successfully (green) and the application of the CL results in a broken (red) build or failed tests Tree Hugger reports a failure.

Build Chaining

In Android, one build depending on the outputs of another build is referred to as a build chain. The Android Rust toolchain makes use of build chaining to

  1. Instrument, profile, and optimize the toolchain
  2. Obtain released NDKs
  3. Test pre-release NDKs
  4. Re-use compilation artifacts to reduce work done by buildbots

When a build target depends on the output of another target in the same branch the job for the dependent target will be scheduled after the dependencies are satisfied. If a build target depends on a target in a different branch it will be built twice. The first time will be in response to the original build trigger and the artifact dependencies will be pulled form the last known good build of the depended upon target. A second build of the target will be triggered when the new build of the depended upon target (launched by the same trigger) is completed, and will use the newly produced artifacts.

The cross-branch chaining behavior can sometimes result in the first of the two builds failing. Specifically, this can happen in the rustc-linux_pgo_bolt build when the BOLT profiles for an older version of the Rust toolchain are provided to a build of newer Rust toolchain source. Check the build log for error messages related to missing or incorrect artifact inputs and then wait for the second build to complete.

Scheduling Pool

When a build is created it must be matched to a buildbot to begin executing. To do this, Launch Control will iterate over the scheduling pool definitions, in order, and find the first pool that matches the branch and target names of the new build.

There are several important concepts to keep in mind when reasoning about and/or modifying the scheduling pools:

  1. An individual buildbot may be a member of more than one scheduling pool.

  2. Exclusive pools prevent the scheduling algorithm from allocating the pinned targets to any pool besides the one marked exclusive: true.

  3. Trident enabled builds need to be mapped to specific Trident scheduling pools that we do not have administrative rights to. Trident is only required for builds that are defined in a Droid Monitored branch and enabled in pre-submit. As none of our branches are Droid Monitored and we want better control over our build scheduling none of our builds should be Trident enabled.

  4. Each scheduling pool only supports a single build_type. As such, separate pools need to be defined for pre- and post-submit builds.

  5. Incremental builds are very finnicky. If any job that is not the same build target from the same branch is mapped to a build bot that previously hosted an incremental build the new build will delete the incremental state. This means that each incremental job must be mapped to a non-shared and exclusive pool whose machines are not members of other pools.

  6. TODO: Add information about default pools when that is sorted out.

TODO: If we are still going to use a custom scheduling pool a description of the pools and the logic behind their allocation should go here.

Branches

Due to https://source.android.com/docs/whatsnew/site-updates#aosp-changes, the Rust toolchain branch is now developed internally in Android. Luckily, we also provide a mirror of our sources using the mirror-goog-main-rust-toolchain-source branch. In the future, we will provide instructions for how to best reproduce our mirrored top-of-tree builds in public.

main-rust-toolchain

The main-rust-toolchain branch is used to generate artifacts for release and thus has a minimal set of post-submit triggers, pre-submit triggers only on the manifest, and no cron triggers.

rust-darwin_mac

A Darwin Rust toolchain

  • Host: darwin-x86
  • Script: tools/build.py
  • Arguments: --llvm-linkage shared --cgu1

rust-linux_glibc

A Linux Rust toolchain built with glibc

  • Host: linux-x86
  • Script: tools/build.py
  • Arguments: --llvm-linkage shared --lto thin --cgu1

rust-linux_musl

A Linux Rust toolchain built with MUSL

  • Host: linux-x86
  • Script: tools/build.py
  • Arguments: --llvm-linkage shared

rust-linux_pgo_bolt

A Linux Rust toolchain built with glibc and optimized using PGO and BOLT profiles from the profiling pipeline

  • Host: linux-x86
  • Script: tools/build.py
  • Arguments: --llvm-linkage shared --lto thin --cgu1 --profile-use <path-to-profiles> --bolt-profile-use <path-to-profiles>

rust-windows_gnu

A cross-compilation build of the Windows toolchain on a Linux host

  • Host: linux-x86
  • Script: tools/build.py
  • Arguments: --llvm-linkage static

rust-windows_gnu_native

The native build of the Windows host toolchain

  • Host: windows-x86
  • Script: tools/build.py
  • Arguments: --llvm-linkage static --cgu1

main-plus-rust

cf_arm64_phone-trunk-userdebug

Build Android's Rust code with the compiler produced by rust-baseline. This target also produces an Android image used for running emulated device tests.

  • Host: linux-x86
  • Script: tools/test_compiler.py
  • Arguments: --all-rust --image --target aosp_cf_arm64_phone --variant userdebug --release trunk

cf_riscv64_phone-trunk-userdebug

Build Android's Rust code with the compiler produced by rust-baseline.

  • Host: linux-x86
  • Script: tools/test_compiler.py
  • Arguments: --image --target aosp_cf_riscv64_phone --variant userdebug --release trunk

cf_x86_64_phone-trunk-userdebug

Build Android's Rust code with the compiler produced by rust-baseline. This target also produces an Android image used for running emulated device tests.

  • Host: linux-x86
  • Script: tools/test_compiler.py
  • Arguments: --all-rust --image --target aosp_cf_x86_64_phone --variant userdebug --release trunk

cf_x86_64_phone-trunk-user

Build Android's Rust code with the compiler produced by rust-baseline. This target tests the user build variant.

  • Host: linux-x86
  • Script: tools/test_compiler.py
  • Arguments: --all-rust --image --target aosp_cf_x86_64_phone --variant user --release trunk

cf_x86_64_phone-trunk-eng

Build Android's Rust code with the compiler produced by rust-baseline. This target tests the eng build variant.

  • Host: linux-x86
  • Script: tools/test_compiler.py
  • Arguments: --all-rust --image --target aosp_cf_x86_64_phone --variant eng --release trunk

pixel-trunk-userdebug

Build Android's Rust code with the compiler produced by rust-baseline. This target also produces an Android image used for running device tests on the most recent in-market Pixel device.

  • Host: linux-x86
  • Script: tools/test_compiler.py
  • Arguments: --all-rust --image --target aosp_husky --variant eng --release trunk

rustdoc

Build all of the Rust documentation in Android.

  • Host: linux-x86
  • Script: tools/test_compiler.py
  • Arguments: --rustdoc --target aosp_cf_x86_64_phone --variant userdebug --release trunk

boltifyer-instrument

Use checked-in resources to add BOLT instrumentation to a Rust toolchain. This helps to verify that our BOLT tooling is working without having to build a fresh toolchain.

  • Host: linux-x86
  • Script: tools/boltifyer.py
  • Arguments: --profile-generate -- resources/rust-1.78.0-relocs.tar.xz

boltifyer-optimize

Use checked-in resources to optimize a Rust toolchain using BOLT. This helps to verify that our BOLT tooling is working without having to build a toolchain, instrument it, and then profile the compiler.

  • Host: linux-x86
  • Script: tools/boltifyer.py
  • Arguments: --profile-use resources/bolt-profiles-1.78.0.tar.xz resources/rust-1.78.0-relocs.tar.xz

merge_profiles-bolt

Use checked-in resources to ensure we can successfully merge BOLT profiles.

  • Host: linux-x86
  • Script: tools/merge_profiles.py
  • Arguments: resources/profiles/cf_arm64 resources profiles/cf_riscv64 resources/profiles/cf_x86_64

merge_profiles-pgo

Use checked-in resources to ensure we can successfully merge PGO profiles.

  • Host: linux-x86
  • Script: tools/merge_profiles.py
  • Arguments: resources/profiles/cf_arm64/bolt-profiles.tar.xz resources profiles/cf_riscv64/bolt-profiles.tar.xz resources/profiles/cf_x86_64/bolt-profiles.tar.xz

rust-baseline

This target builds a Linux Rust toolchain as a method of testing the toolchain's build system. The resulting compiler is then used by other build targets in this branch to ensure that it produces correct output.

  • Host: linux-x86
  • Script: tools/build.py
  • Arguments: --llvm-linkage shared --lto thin

rust-linux_ndk

A canary build for the NDK top-of-tree builds. This build is meant to alert us to problems in upcoming NDK releases before they occur.

  • Host: linux-x86
  • Script: tools/build.py
  • Arguments: --llvm-linkage shared --lto thin --no-bare-targets --no-host-multilibs

test_suites_x86_64

This build produces artifacts for running host and device tests. The build target definition is a copy of the target of the same name in main.

  • Host: linux-x86
  • Script: <android root>/build/soong/soong_ui.bash
  • Arguments: --make-mode cts dist DIST_DIR=%dist_dir% TARGET_PRODUCT=aosp_x86_64 TARGET_RELEASE=trunk_staging WITH_DEXPREOPT_BOOT_IMG_AND_SYSTEM_SERVER_ONLY=true acts_tests vts tradefed-all general-tests test_mapping robolectric-tests host-unit-tests

main-rust-profiling

This branch is responsible for generating PGO and BOLT profiles for use by the release branch. A build is scheduled once a week on the weekends via cron as well as whenever there are commits to toolchain/rustc. If you need to manually trigger the creation of new profiles you can create an empty commit to the toolchain/rustc project.

The build targets in each stage depend on the output of the previous stage, with the stage2-merge and stage5-merge build targets acting as merge points for the stages with multiple targets. A full run of this pipeline can take over six hours to complete.

stage1-rust-pgo-inst

Build a Linux Rust toolchain with PGO instrumentation.

  • Host: linux-x86
  • Script: tools/build.py
  • Arguments: --llvm-linkage shared --lto thin --cgu1 --profile-generate

stage2-cf_arm64_phone-trunk-userdebug

Use the toolchain from stage1-rust-pgo-inst to generate PGO profiles while compiling Android Rust targets for cf_arm64.

  • Host: linux-x86
  • Script: tools/test_compiler.py
  • Arguments: --all-rust --profile-generate --target aosp_cf_arm64_phone --variant userdebug --release trunk

stage2-cf_riscv64_phone-trunk-userdebug

Use the toolchain from stage1-rust-pgo-inst to generate PGO profiles while compiling Android Rust targets for cf_riscv64.

  • Host: linux-x86
  • Script: tools/test_compiler.py
  • Arguments: --image --profile-generate --target aosp_cf_riscv64_phone --variant userdebug --release trunk

stage2-cf_x86_64_phone-trunk-userdebug

Use the toolchain from stage1-rust-pgo-inst to generate PGO profiles while compiling Android Rust targets for cf_x86_64.

  • Host: linux-x86
  • Script: tools/test_compiler.py
  • Arguments: --all-rust --profile-generate --target aosp_cf_x86_64_phone --variant userdebug --release trunk

stage2-merge

Merge profiles from the stage2-cf_* targets into a single profile.

  • Host: linux-x86
  • Script: tools/merge_profiles.py
  • Arguments: <list of paths to profiles>

stage3-rust-pgo-opt

Build a Linux Rust toolchain using the profile from stage2-merge to preform profile-guided optimization. This toolchain contains the relocation information required by BOLT in later stages.

  • Host: linux-x86
  • Script: tools/build.py
  • Arguments: --llvm-linkage shared --lto thin --cgu1 --profile-use --emit-relocs

stage4-rust-bolt-inst

Add BOLT instrumentation to the Rust toolchain generated by stage3-rust-pgo-opt.

  • Host: linux-x86
  • Script: tools/boltifyer.py
  • Arguments: --profile-generate

stage5-cf_arm64_phone-trunk-userdebug

Use the toolchain from stage4-rust-bolt-inst to generate BOLT profiles while compiling Android Rust targets for cf_arm64.

  • Host: linux-x86
  • Script: tools/test_compiler.py
  • Arguments: --all-rust --bolt-profile-generate --target aosp_cf_arm64_phone --variant userdebug --release trunk

stage5-cf_riscv64_phone-trunk-userdebug

Use the toolchain from stage4-rust-bolt-inst to generate BOLT profiles while compiling Android Rust targets for cf_riscv64.

  • Host: linux-x86
  • Script: tools/test_compiler.py
  • Arguments: --image --bolt-profile-generate --target aosp_cf_riscv64_phone --variant userdebug --release trunk

stage5-cf_x86_64_phone-trunk-userdebug

Use the toolchain from stage4-rust-bolt-inst to generate BOLT profiles while compiling Android Rust targets for cf_x86_64.

  • Host: linux-x86
  • Script: tools/test_compiler.py
  • Arguments: --all-rust --bolt-profile-generate --target aosp_cf_x86_64_phone --variant userdebug --release trunk

stage5-merge

Merge profiles from the stage5-cf_* targets into a single profile.

  • Host: linux-x86
  • Script: tools/merge_profiles.py
  • Arguments: <list of paths to profiles>

stage6-rust-bolt-opt

Use the BOLT profiles from stage5-merge to optimize the Rust toolchain generated by stage3-rust-pgo-opt.

  • Host: linux-x86
  • Script: tools/boltifyer.py
  • Arguments: --profile-use

stage7-cf_arm64_phone-trunk-userdebug

Verify that the compiler produced by stage6-rust-bolt-opt can compile the Android Rust codebase.

  • Host: linux-x86
  • Script: tools/test_compiler.py
  • Arguments: --all-rust --target aosp_cf_arm64_phone --variant userdebug --release trunk