documents/RELEASE.md

# Android Rust Toolchain Release Process

The upstream Rust project releases a new stable version of the Rust toolchain
every 6 weeks.  Android's Rust toolchain team **must** release a prebuilt for
every toolchain version as new compilers often depend on features released in
the previous version.  This means that skipping a release would inevitably
result in a failure when we attempted to update the toolchain in the future.

A high-level overview of the steps necessary to release a new Rust toolchain
for Android can be found in the [following section](#task-checklist) with more
detailed instructions appearing in the [Release Process](#release-process)
section.

General Android development is outside the scope of this document and it is
expected that readers have a working knowledge of
Soong
[[1](https://ci.android.com/builds/submitted/11792654/linux/latest/view/soong_build.html),
[2](https://docs.google.com/document/d/1SX-49NEdQaktfbmyWEwkl0TBukrxE3bnvideoKK9jMw/edit?usp=sharing)],
[`repo`](https://source.android.com/docs/setup/reference/repo),
[`lunch`](https://source.android.com/docs/setup/build/building#choose-a-target),
and how to
[modify Android code](https://g3doc.corp.google.com/company/teams/android/developing/source-control.md?cl=head).
Information on how to debug issues that occur while conducting the release
process can be found in the [debugging](./DEBUGGING.md) documentation.

## Rust Version Strings

The Rust project uses [semantic versioning](https://semver.org/) for its
software releases.  As such, their version strings follow the
`<major>.<minor>.<patch>` format.  If a "point release" (re-compilation and
distribution of the same source code built with different arguments) is
performed for the Rust toolchain any new version string must adhere to the
following format: `<major>.<minor>.<patch>.<point>` (E.g. `1.77.1.p1`).

## Task Checklist

Note: Copy this Markdown source into the update's issue's `Description` field.

- [ ] Fetch Rust source
- [ ] Obtain updated patches from Trusty
- [ ] Update build system
- [ ] Update platform Rust code for new compiler version
- [ ] Update internal Rust code for new compiler version
- [ ] Upload new Rust source and build system changes
- [ ] Upload new Rust prebuilts
- [ ] Notify [developers and dependent teams](g/android-rust-announce) about toolchain release
- [ ] Create Issue for next release

## Release Process

The steps listed below may contain invocations of the project's custom scripts.
Documentation for these tools can be found in [here](./DEVELOPMENT.md) as well
as in the tools themselves via the `--help` flag.

It is **highly recommended** to begin the update process with a freshly
synchronized Android source tree.

### Fetching Rust Source

Begin by using the [`fetch_source.py`](./DEVELOPMENT.md#fetch-source) tool to
download an archive containing the Rust toolchain source.  By default it will
fetch the release archive for the provided Rust version string.  The `--beta`
and `--nightly` flags can be used to fetch their respective source archives.

The `fetch_source.py` will occasionally fail to commit the new source code,
with a message warning of prohibited patterns in the commit.  The offending
texts are keys used in CI-related configurations, not actual private keys.

These warnings can be suppressed by running the following command in the
`toolchainrustc` directory:
`git config --add secrets.allowed ARTIFACTS_AWS_ACCESS_KEY_ID`

Note: It is helpful to use the same Git branch name in the
`toolchain/android_rust` project for the next two steps.  This will allow you to
use the `-t` argument to set the topic of the CLs when you upload them.

### Recontextualizing the Patches

As the Rust source code changes our local patches will drift out of context or
break outright.  The
[`recontextualize_patches.py`](./DEVELOPMENT.md#recontextualize-patches) tool
is used to reduce the former and help fix the latter.  After fetching the new
source run this tool.  If everything is successful you can commit the updated
patch files in the `toolchain/android_rust` project and move to the next step.

If there was an issue with the patch update process the tool will exit with an
error.  You can then navigate to `toolchain/rustc` and finish the `git am`
process that is in progress.  You can use `git` or `patch` to re-apply the
patch that failed, resolve the issues, commit the changes, and then run
`git am --continue`.

When the `git am` process is finished you can run
`tools/recontextualize_patches.py --resume` to copy the updated patches into
`toolchain/android_rust/patches`.

### Updating the Build System

It is sometimes necessary to modify the Android Rust toolchain build system
to successfully build a new version of Rust.  The types of changes necessary
can vary from adding a new value to the `config.toml.template` file, to
investigating the source of a new shared library dependency, to setting
environment variables to enable/disable new features.

Start by using the following command to initiate a build of the new Rust
source: [`./tools/build.py --lto thin --llvm-linkage shared`](./DEVELOPMENT.md#build)

See the [debugging](./DEBUGGING.md) document for tips and suggestions on fixing
build breakages.  To avoid repeatedly copying and/or patching the source code
during development you can use the `--repatch` and `--no-copy-and-patch`
arguments.

Use the `--upstream-test` flag to run the upstream Rust project's host tests.

Once you have fixed any build breakages you can create a new Git commit on top
of the existing update to the patches.

### Updating Vendored Package Versions

The `vendor/` directory of the Rust release archives contain multiple versions
of many crates.  There are several of these crates that we must copy into the
Android toolchain's prebuilt directory for use by Soong.  Because Soong does
not support having multiple versions of a project's source we must pick which
of the multiple versions in the release archive we wish to use.

During an update it is necessary to check each of the vendored crates defined
in [`STDLIB_SOURCE_RENAMES`](../tools/build.py#L79) and update the version
number if necessary.  If there are multiple versions of a crate available
select the most recent one for initial testing.  If there is a build failure
when building the crate with Soong you can copy the next candidate version of
the crate directly into the prebuilt's directory (e.g.
`prebuilts/rust/linux-x86/1.82.0/src/stdlibs/vendor/getopts`) and attempt to
rebuild the target.

### Updating Android

Once you have build a Rust toolchain you can begin the process of updating the
Android source code to be compatible with the new compiler.  Use the
[`test_compiler.py`](./DEVELOPMENT.md#test-compiler) tool to extract the
prebuilt archive and begin running the tests.

When a target breaks resolve the issue to the best of your ability.  The
diagnostic message from the compiler will often be enough to address the error.
If you are uncertain of what changes to make to an internal Android project you
can add a comment with the text of the error, upload the CL, and contact the
code owners for advice on the issue.

After resolving an issue you can resume the testing process by either passing
the `--reuse-prebuilt` argument to the `test_compiler.py` tool or executing
`export RUST_PREBUILTS_VERSION=9.99.9` before manually invoking Soong to build
the test targets (e.g. `m rust`).

Upload any changes required by the new compiler version to Gerrit for review.
Changes that are valid in both the old and new version of Rust should go in
independently while any changes that require the new compiler will need to go
into the topic used to update the prebuilts.

These changes need to go in before the compiler source is updated to ensure that
the CI builds remain green.

Note: An [example](https://b.corp.google.com/issues/270756468) of a ticket that
posted all related CLs in the `Description`.

#### Booting an Android Image

If the `--image` flag was passed to `test_compiler.py` the resulting image
(default target `aosp_cf_x86_64_phone-trunk_staging-userdebug`) can be used to
boot a device.  A virtual device can be created using the following command:

```shell
$ acloud create --local-image
```

Once it is complete, this command will print the IP address and port that can be
used to connect to the virtual device. Export this is as `ANDROID_SERIAL` or use
it as a command line argument to Android development tools.

Use the `--local-instance` flag to run the virtual device on the machine running
the command. This is [not recommended](https://g3doc.corp.google.com/security/g3doc/eip/ps/consulting-docs/hyperthreading.md)
for Google developers.

### Uploading New Compiler Source

The Android CI infrastructure triggers new builds of the compiler either
`toolchain/android_rust` or `toolchain/rustc` are updated.  This compiler is
then used to invoke `test_compiler.py`.  As such, if the Android source hasn't
been updated for the new compiler the new compiler source will fail to pass
presubmit checks.

Because of this dependency you will need to wait until all CLs updating the
Android source code for the new compiler have been merged.  Once that has
occurred you can upload the CLs containing the recontextualized patches, any
required build system changes, and the new Rust source.

These CLs will all need to be tested together, which requires that they are
members of the same topic in Gerrit.  The topics can be set manually or you can
pass the `-t` flag to `repo` to use the Git branch name as the topic.

If `repo` complains about private keys you can pass the `-o nokeycheck` option
to skip they key check.

### Uploading New Prebuilts

Android's Rust prebuilt toolchains are compiled using PGO and BOLT.  The
profiles for these optimizations are generated by the build targets in the
[`git_main-rust-profiling`](./BUILDS.md#git_main-rust-profiling) branch.  After
uploading the source for a new version of Rust you will need to wait for two
builds to finish: 1) the first successful build of
[`stage5-merge`](./BUILDS.md#stage5-merge) in `git_main-rust-profiling` and 2) the
first successful build of
[`rust-linux_pgo_bolt`](./BUILDS.md#rust-linux_pgo_bolt) after the completion
of the pervious `stage5-merge` build.  This will ensure that the new prebuilts
are compiled with the most recent prebuilts.

BOLT profiles are specific to a particular file name.  Because Rust libraries
include a checksum in their file names BOLT profiles from one version of the
toolchain will fail to apply to a new version.  This will result in at least
one broken `rust-linux_pgo_bolt` built after an update, as
[build chaining](./BUILDS.md#build-chaining) will start an initial build of
the target using the profiles from the last known good build of `stage5-merge`.

Once profiles have been generated and a new toolchain built the artifacts need
to be fetched, unpacked, and checked in.  In addition, the Rust toolchain and
Soong build systems need to be updated with the new compiler version number.
The [`update_prebuilts.py`](./DEVELOPMENT.md#update-prebuilts) script handles
each of these steps, creating commits in the `build/soong`, `prebuilts/rust`,
and `toolchain/android_rust` projects.

For a normal release the following command can be used:
```shell
$ ./tools/update_prebuilts.py -i <issue number> <rust version>
```

If the automatic _last known good build_ detection fails you can use the
`--bid` flag to manually select a build ID to fetch artifacts from.

Upload the three commits crated by the tool and place them in a single topic
(e.g. `rust-update-source-1.78.0`.) before proceeding with the review process.

### Notification

Once the new prebuilts have been submitted send an email to the
`android-rust-announce@google.com` mailing list.  There is no standard template
for these announcements, but including a link to the issue page or the release
notes can be helpful.

### Cleanup

Create an issue for the next Rust toolchain release and assign it to the next
engineer in the release rotation.

It is a goal of the Android Rust team to only keep
[in-use toolchains](./PREBUILTS.md) in the prebuilts directory as any previous
toolchains can be recovered using Git.  After an update it can take downstream
teams several days to weeks to complete their own update processes.  During
this time a CL can be created to delete the previous version of the toolchain
but it can not be submitted until all presubmit tests pass and it has been
confirmed that downstream teams have updated their build systems as necessary.

If a downstream team takes longer than a week to complete their update process
it might be appropriate, after discussion with the client team, to pin their
manifest to the current commit before proceeding with removing the old
perbuilts.

Example commit message:

```
Removing Rust $OLD_RUST_VERSION prebuilts

Bug: <bug ID>
Test: m rust
```

Prebuilts exist for the following host architectures:
* `darwin-x86`
* `linux-x86`
* `linux-musl-x86`
* `windows-x86`

Once the update is complete notes should be added to the
[update log](https://docs.google.com/document/d/1FQmVZXiCRIwkMOXnpClCDJI4ZVxbZUbe4RKZ6MgO468/edit)
detailing any issues encountered or changes made.

## Beta Testing

The Rust project will publish beta releases in the weeks and days leading up to
a major release.  It can often be beneficial to use these beta releases for
testing and to begin the update process.

The `fetch_source.py` tool can be used to download the beta archives using the
`--beta` flag and then built normally with `build.py`.  While the beta source
can not be submitted, build system changes and patch updates can be prepared.
If a beta compiler is successfully built and produces errors when compiling the
Android codebase fixes for these errors can be submitted immediately, as long
as the current compiler considers them valid.

Making changes based on a beta release too early can be detrimental to the
release process, however.  Patches can be reverted from the beta branch leading
to large, last minute source code changes or feature removals.  This could
result in a need to re-recontextualize the patches or undo changes to the build
system.  It is therefore **not** recommended to test on a beta source archive
more than 1 or 2 weeks in advance of a release.  Lastly, it is often better to
avoid beta testing in the days immediately prior to an official release and
instead wait for the new version of the toolchain source to become available.