documents/DEBUGGING.md

While updating the Rust toolchain there are various issues that can arise.
Things can break, roadblocks can get in the way, and others might need to be
brought in. In this section we describe the different types of issues that can
occur, examples, and instruct on how to move past them.

They are organized by topics: - Section 2.1: Itemizes various breaks that can
occur when building Rust during Step Test-2 *Build locally*. - Section 2.2:
Describes roadblocks hit when testing during Step Test-3:*Test*, which is a part
of Step Release-8: *Testing Build* - Section 2.3: Proves some notes on updating
the prebuilts during Step Release-9:*Update Prebuilts* - Section 2.4: Describes
issues with uploading to Gerrit as done in both Step Release 6:*Upload* and Step
Release 10:*Upload to Gerrit*

# The Rust Build

Things that can break while building the Rust toolchain:

-   Patch Application
-   Directory Structure Change
-   Binary Incompatibility
-   Compilation Failure
-   New Crate
-   New flags
-   Missing tools

## Patch Application

The Android project carries several patches for the Rust toolchain source.
Patches make changes to a code file. In order for a patch file to be
successfully applied the code that it is targeting needs to match the code file
closely enough for the algorithm in the `patch` program to identify the relevant
code. If a patch file fails to be applied then it is likely due to a change in
the targeted code base. In which case, the next step is to figure out if the
patch is still necessary to be applied or if that Patch file can be removed.

To know which patch file failed take a look at the terminal error message. The
error message will also say what *hunk #* and name of the Rust file. Open up the
patch file in `android_rust/patches` and the Rust file. At this point it'll
either look like (1) patch was already applied, (2) the codebase was otherwise
modified.

In the case of (1), it would be useful to verify that the patch was applied.
This can be done by look at the log history for that Rust file.If you do believe
that the patch was merged upstream then you just need to remove the patch from
the `patches/` directory, e.g.

```
pushd toolchain/android_rust
repo start update-rustc-$RUST_VERSION
git rm patches/rustc-000n-Already-merged.patch
git commit -m "Remove Foo patch that has landed upstream"
popd
```

In the case of (2), the codebase was changed in some way. Sometimes the
difference might be very simple. For instance, one time the difference was just
a variable name change and looking at the log history confirmed that.

To be able to successfully apply the patch the code in the patch must match the
code base exactly so go ahead and modify the patch code to match the code base.
If adding or removing lines of code be sure to update the number of lines that
is noted in the patch file. If you are unsure if the change upholds the intent
of the patch go ahead and email the patch owner, but note they will also be
added as a reviewer.  When generating patch files using `git format-patch` be
sure to include extra context (`-U10`) as it is needed to successfully apply
some patches.

After editing the patch file, upload the change to Gerrit, and ask the patch
owner to review the changes to make sure the intent of the patch is still
Upheld. Use the topic with "source".

Here is an example of updating a patch file to correspond to changes in code
[CL](https://android-review.googlesource.com/c/toolchain/android_rust/+/1999334).

It may also be useful to create a new patch. The following:

```
git format-patch -U10 HEAD~
```

will generate a patch file for just the previous commit.

## Directory Structure Change

Sometimes another library needs to be imported. We can do this by adding to the
`STDLIB_SOURCES` definition in the `tools/build.py` script.

For example we got the following:

```shell
error: couldn't read prebuilts/rust/linux-x86/1.59.0/src/stdlibs/library/core/src/../../portable-simd/crates/core_simd/src/mod.rs: No such file or direct
ory (os error 2)
   --> prebuilts/rust/linux-x86/1.59.0/src/stdlibs/library/core/src/lib.rs:415:1
    |
415 | mod core_simd;
    | ^^^^^^^^^^^^^^
error: aborting due to previous error

22:37:34 ninja failed with: exit status 1

#### failed to build some targets (18 seconds) ####

```

and as a result we added the portable-simd library as seen in this
[CL](https://android-review.googlesource.com/c/toolchain/android_rust/+/1999334/4/do_build.py).

## Binary Incompatibility

*TODO: text here to describe how the user will know this type of error occurred
and how to fix it*

## Compilation Failure

*TODO: text here to describe how the user will know this type of error occurred
and how to fix it*

## New Crate

Sometimes a new crate is added and a modification needs to be made to
`prebuilts/rustc/Android.bp`.

*TODO: text here to describe how the user will know this type of error occurred
and how to fix it*

## New Flags

The solution might be to disable a new flag that have been added to the build
process.
[Example CL](https://android-review.googlesource.com/c/toolchain/android_rust/+/2103672/3/templates/config.toml.template)

Check the `out` file `CMakeCache.txt` to see any additional flags that may have
been added during the build process. Try testing the new flags with the older
version of the compiler.

## Missing Tools

```
failed to execute command: "strip" "--strip-debug" "/usr/local/google/home/aosp/out/rustc/build/x86_64-unknown-linux-gnu/llvm/lib/libLLVM-17.so"
error: No such file or directory (os error 2)
```

This indicates that `strip` is not available on your local computer. We solved
this by adding a symbolic link to another tool
[CL](https://android-review.git.corp.google.com/c/toolchain/android_rust/+/2776808/8/build.py)

## Troubleshooting a Broken Sysroot Build

*Question: Should this be added to `Section 2.1 Build Error - Missing Crate` or
is it a different type of error?*

If the sysroot build is broken, check whether the error mentions a missing
crate. If it does, there have likely been new components added to the sysroot.
To address this, you will need to:

1.  Add the relevant components to `STDLIB_SOURCES` in
    `toolchain/android_rust/tools/build.py`.
2.  Respin the toolchain via the process above, but with a fresh commit message
    noting the reason for the respin. You may want to test this locally first as
    more than one dependency may have been added. For local testing,
    1.  Build as before, using `DIST_DIR=$TOOLCHAIN/dist
        ./toolchain/android_rust/tools/build.py`
    2.  Make a local commit with the contents of the tarball now in `$DIST_DIR`
    3.  Go to `prebuilts/rust` in your Android tree and use `git fetch local`
        followed by a checkout to get it imported into your Android tree.
3.  Add the missing dependencies to `prebuilts/rust/Android.bp`. Except for
    publicly exported crates (which you're not adding right now), all modules in
    this file must be suffixed with `.rust_sysroot` to avoid confusion with user
    crates. Dependency edges should all be of `rlib` form unless depending on a
    publicly exported crate, in which case the dependency edge should match the
    type of the final product. As examples, `libterm` (exported) depends by
    `dylib` on `libstd`, but `libterm.static` (also exported) depends by `rlib`
    on `libstd.static`.`libhashbrown.rust_sysroot` is built only as an `rlib`,
    and is linked as an `rlib` everywhere it is used.

# Test

Things that can break during testing:

-   Hermeticity breakage
-   Build system breakage
-   Android Source Warnings
-   Miscompilation
-   Deprecated Flag

## Hermeticity Breakage

A Hermeticity breakage occurs when the build uses a tool or header that is
different from what we expect. For instance, a crate including a build script
allowing a different compiler/library to be used than expected. Some of these
issues arise because of bugs in the upstream build system. Others can also be
caused because the environment they are built-in have certain defaults set and
carry unintended expectations.

*TODO* @Chriswailes: Describle how to approach this type of problem, given
lzma-sys example.

## Build System Breakage

There can be build breakage issues.

For instance, needing to change `"-C passes='sancov'"`, to `"-C
passes='sancov-module'"`, such as in this
[CL](https://android-review.googlesource.com/c/platform/build/soong/+/2003172/4/rust/sanitize.go).

## Android Source Warnings

A common issue is that this step triggers new warnings on existing source files.
If the compiler suggests a fix, apply it. Otherwise make the most reasonable
looking change necessary to keep the compiler happy and rely on the original
author to determine correctness during code review.

Here is an example of the workflow to modify file x.rs:

```
# navigate to the repo with file x.rs
repo start rust-1.59.0-fixes
# make changes to x.rs
git add x.rs
git commit
```

The commit message should include the testing strategy and buganizer ticket
number. Here is an example commit message from one of these types of
[CLs](https://android-review.googlesource.com/c/platform/system/security/+/2002316).

```
Changes for the Rust 1.59.0 update

bug: 215232614
Test: TreeHugger and compiling with m rust
Change-Id: I1d25f5550f60ff1046a3a312f7bd210483bf9364
```

Note, that it is preferable to create one commit per big change to a repo, so it
might be helpful to use amend when adding more changes to the code:

```shell
$ git commit --amend --no-edit
```

After committing the changes continue to upload them to Gerrit with

```shell
repo upload .
```

In Gerrit for the corresponding CL include the owners of the file as reviewers
and do not set a `topic`.

Next we will need to periodically check-in on these CLs. If they pass presubmit,
then we are waiting for the CLs to be approved/submitted by the file owners.
This can take a few days and may require a friendly nudge.

If the files do not pass presubmit then the changes may not have been backwards
compatible with Rust and we will need to compile it with the latest version of
Rust. If that is the case on Gerrit include it in the topic
`rust-update-prebuilts-1.59.0`, with an updated Rust version number.

We are not able to move to the final **Step 12** (tagging) until these CLs
created in this process has been submitted/accepted.

Clipper errors are typically pretty straightforward but sometimes it’s not clear
what to change. Try checking at the change history or how the change has been
implemented in other parts of the Rust codebase. For instance,
`builder.config.llvm_link_shared` was changed to `!builder.llvm_link_shared()`.

## Miscompilation

A miscompilation may have occured when it successfully compiles but the devices
fail to boot or pass CTS tests.

*TODO: text here to describe how the user will know this type of error occurred
and how to fix it*

## Deprecated flag

The following error:

```shell
warning: `-Z symbol-mangling-version` is deprecated; use `-C symbol-mangling-version`
```

led to changes where that variable was used with `-Z` was changed to `-C` in
`rust/config/global.go`.
[CL](https://android-review.googlesource.com/c/platform/build/soong/+/2003172/5/rust/config/global.go)
.

## Delete files

With this error
```
~/updates/aosp
: > ./toolchain/android_rust/tools/test_compiler.py --prebuilt-path dist/rust-dev.tar.xz --target aosp_cf_x86_6
4_phone --image
Test prebuilt directory already exists.  Deleting contents.
Extracting archive /usr/local/google/home/chiw/updates/aosp/dist/rust-dev.tar.xz
/usr/local/google/home/chiw/updates/aosp/prebuilts/build-tools/linux-x86/bin/xz: /usr/local/google/home/c
hiw/updates/aosp/dist/rust-dev.tar: File exists
Traceback (most recent call last):
  File "/usr/local/google/home/chiw/updates/aosp/./toolchain/android_rust/test_compiler.py", line 198, in
 <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/google/home/chiw/updates/aosp/./toolchain/android_rust/test_compiler.py", line 143, in
 main
    prepare_prebuilts(args.prebuilt_path)
  File "/usr/local/google/home/chiw/updates/aosp/./toolchain/android_rust/test_compiler.py", line 129, in
 prepare_prebuilts
    (target_and_version_path / "bin" / "rustc").touch()
  File "/usr/lib/python3.11/pathlib.py", line 1109, in touch
    fd = os.open(self, flags, mode)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/google/home/chiw/updates/aosp/prebuil
ts/rust/linux-x86/9.99.9/bin/rustc
```
Try
```
rm $DIST_DIR/rust-dev.tar
```

## New Build Breaking Lint/Clippy Errors

New lints/clippys can cause build breakage and may require significant
refactoring as the code base grows. To avoid blocking toolchain upgrades,
explicitly allow the breaking lints/clippys when first upgrading the toolchain.

1.  Allow build breaking lints/clippys by adding them to the list in
    `build/soong/rust/config/lints.go` with the `-A` flag.
2.  If the new lint/clippy is beneficial enough to justify enable going forward,
    file a bug to track the refactor effort.

# Update prebuilts

Things that can break when updating the prebuilts:

-   File undefined
-   Fetch ID
-   Unknown manifest

## File undefined

Try deleting the branch `git branch -D rust-update-prebuilts-1.59.0-local`

## Fetching ID

```shell
: > ./toolchain/android_rust/tools/update_prebuilts.py
--chained 1.62.0 Unable to fetch LKGB build ID
```
This means you forgot to run gcert.

## Unknown Manifest

Note that some tests might fail during presubmit because of code that live in
other manifests. Make the appropriate changes as usual, but in the right place.

# Upload

Things that can break while uploading files to Gerrit:

-   Geritt limitation
-   Detached head

## Geritt Limitation

Help, Gerrit won't take my update!

First, try again. Sometimes Gerrit is just flaky and will take it on the second
or third try.

If that's still not working, you are likely hitting a size limitation (for
example, because `rustc` updated it's LLVM revision, so the diff is bigger than
usual). In this case, you will need to work with the build team to get them to
use a "direct push" to skip Gerrit's hooks. Look at the initial import
[bug](http://b/137197907) for an example conversation about importing oversized
changes.

## Detached head

Use this to get away from detached head:

```shell
git branch -u aosp/master
```

# Prebuilts Test failure

*   Run tests locally
*   Output Format

## Run test locally

If presbuilts failed presubmit, one thing you can do is try to run test locally

```shell
source build/envsetup.sh
lunch aosp_cf_x86_64_phone-staging-userdebug
m
acloud create --local-image
export ANDROID_SERIAL=<value printed by previous command>
atest bytes_test_tests_test_bytes
// or whatever tests are failing in the prebuilts
```

## Output Format

There could be an error parsing the results of running Rust tests.

A prior release Rust v.1.61 release notes include messages with ignored tests.
We had to modify how the trade federation code analyzes the result of running
tests (changed regex).
[CL](https://android-review.googlesource.com/c/platform/tools/tradefederation/+/2110829)
The Gerrit CL triggered the same changes in Critique.

# Other Helpful Debugging Tips

## Toolchain scripts

Sometimes it might be necessary to change the *android_rust* scripts
themselves. Here are some examples:

*   audit.py: Add more .so files to the shared library allow list because they
    are required by other .so files
*   build.py: Useful to edit if you want to regenerate a lockfile, exclude a
    test
*   config.py: Change env rust flags

## Other

* build_error.log: Check the state of the builds and try to reproduce
locally. For example if Soongs prints `out/a` on failure then try running `m
out/a` to reproduce the issue locally. *
[upstream](https://github.com/rust-lang/rust/tree/beta) Check recent relevant
changes to the Rust upstream code since the last update