dexpreopt/DEXPREOPT_IMPLEMENTATION.md

## Dexpreopt implementation

### Introduction

All dexpreopted Java code falls into three categories:

- bootclasspath
- system server
- apps and libraries

Dexpreopt implementation for bootclasspath libraries (boot images) is located in
[soong/java] (see e.g. [soong/java/dexpreopt_bootjars.go]), and install rules
are in [make/core/dex_preopt.mk].

Dexpreopt implementation for system server, libraries and apps is located in
[soong/dexpreopt]. For the rest of this section we focus primarily on it (and
not boot images).

Dexpeopt implementation is split across the Soong part and the Make part. The
core logic is in Soong, and Make only generates configs and scripts to pass
information to Soong.

### Global and module dexpreopt.config

The build system generates a global JSON dexpreopt config that is populated from
product variables. This is static configuration that is passed to both Soong and
Make. The `$OUT/soong/dexpreopt.config` file is generated in
[make/core/dex_preopt_config.mk]. Soong reads it in [soong/dexpreopt/config.go]
and makes a device-specific copy (this is needed to ensure incremental build
correctness). The global config contains lists of bootclasspath jars, system
server jars, dex2oat options, global switches that enable and disable parts of
dexpreopt and so on.

The build system also generates a module config for each dexpreopted package. It
contains package-specific configuration that is derived from the global
configuration and Android.bp or Android.mk module for the package.

Module configs for Make packages are generated in
[make/core/dex_preopt_odex_install.mk]; they are materialized as per-package
JSON dexpreopt.config files.

Module configs in Soong are not materialized as dexpreopt.config files and exist
as Go structures in memory, unless it is necessary to materialize them as a file
for dependent Make packages or for post-dexpreopting. Module configs are defined
in [soong/dexpreopt/config.go].

### Dexpreopt in Soong

The Soong implementation of dexpreopt consists roughly of the following steps:

- Read global dexpreopt config passed from Make ([soong/dexpreopt/config.go]).

- Construct a static boot image config ([soong/java/dexpreopt_config.go]).

- During dependency mutator pass, for each suitable module:
    - add uses-library dependencies (e.g. for apps: [soong/java/app.go:deps])

- During rule generation pass, for each suitable module:
    - compute transitive uses-library dependency closure
      ([soong/java/java.go:addCLCFromDep])

    - construct CLC from the dependency closure
      ([soong/dexpreopt/class_loader_context.go])

    - construct module config with CLC, boot image locations, etc.
      ([soong/java/dexpreopt.go])

    - generate build rules to verify build-time CLC against the manifest (e.g.
      for apps: [soong/java/app.go:verifyUsesLibraries])

    - generate dexpreopt build rule ([soong/dexpreopt/dexpreopt.go])

- At the end of rule generation pass:
    - generate build rules for boot images ([soong/java/dexpreopt_bootjars.go],
      [soong/java/bootclasspath_fragment.go] and
      [soong/java/platform_bootclasspath.go])

### Dexpreopt in Make - dexpreopt_gen

In order to reuse the same dexpreopt implementation for both Soong and Make
packages, part of Soong is compiled into a standalone binary dexpreopt_gen. It
runs during the Ninja stage of the build and generates shell scripts with
dexpreopt build rules for Make packages, and then executes them.

This setup causes many inconveniences. To name a few:

- Errors in the build rules are only revealed at the late stage of the build.

- These rules are not tested by the presubmit builds that run `m nothing` on
  many build targets/products.

- It is impossible to find dexpreopt build rules in the generated Ninja files.

However all these issues are a lesser evil compared to having a duplicate
dexpreopt implementation in Make. Also note that it would be problematic to
reimplement the logic in Make anyway, because Android.mk modules are not
processed in the order of uses-library dependencies and propagating dependency
information from one module to another would require a similar workaround with
a script.

Dexpreopt for Make packages involves a few steps:

- At Soong phase (during `m nothing`), see dexpreopt_gen:
    - generate build rules for dexpreopt_gen binary

- At Make/Kati phase (during `m nothing`), see
  [make/core/dex_preopt_odex_install.mk]:
    - generate build rules for module dexpreopt.config

    - generate build rules for merging dependency dexpreopt.config files (see
      [make/core/dex_preopt_config_merger.py])

    - generate build rules for dexpreopt_gen invocation

    - generate build rules for executing dexpreopt.sh scripts

- At Ninja phase (during `m`):
    - generate dexpreopt.config files

    - execute dexpreopt_gen rules (generate dexpreopt.sh scripts)

    - execute dexpreopt.sh scripts (this runs the actual dexpreopt)

The Make/Kati phase adds all the necessary dependencies that trigger
dexpreopt_gen and dexpreopt.sh rules. The real dexpreopt command (dex2oat
invocation that will be executed to AOT-compile a package) is in the
dexpreopt.sh script, which is generated close to the end of the build.

### Indirect build rules

The process described above for Make packages involves "indirect build rules",
i.e. build rules that are generated not at the time when the build system is
created (which is a small step at the very beginning of the build triggered with
`m nothing`), but at the time when the actual build is done (`m` phase).

Some build systems, such as Make, allow modifications of the build graph during
the build. Other build systems, such as Soong, have a clear separation into the
first "generation phase" (this is when build rules are created) and the second
"build phase" (this is when the build rules are executed), and they do not allow
modifications of the dependency graph during the second phase. The Soong
approach is better from performance standpoint, because with the Make approach
there are no guarantees regarding the time of the build --- recursive build
graph modfications continue until fixpoint. However the Soong approach is also
more restictive, as it can only generate build rules from the information that
is passed to the build system via global configuration, Android.bp files or
encoded in the Go code. Any other information (such as the contents of the Java
manifest files) are not accessible and cannot be used to generate build rules.

Hence the need for the "indirect build rules": during the generation phase only
stubs of the build rules are generated, and the real rules are generated by the
stub rules during the build phase (and executed immediately). Note that the
build system still has to add all the necessary dependencies during the
generation phase, because it will not be possible to change build order during
the build phase.

Indirect buils rules are used in a couple of places in dexpreopt:

- [soong/scripts/manifest_check.py]: first to extract targetSdkVersion from the
  manifest, and later to extract `<uses-library/>` tags from the manifest and
  compare them to the uses-library list known to the build system

- [soong/scripts/construct_context.py]: to trim compatibility libraries in CLC

- [make/core/dex_preopt_config_merger.py]: to merge information from
  dexpreopt.config files for uses-library dependencies into the dependent's
  dexpreopt.config file (mostly the CLC)

- autogenerated dexpreopt.sh scripts: to call dexpreopt_gen

### Consistency check - manifest_check.py

Because the information from the manifests has to be duplicated in the
Android.bp/Android.mk files, there is a danger that it may get out of sync. To
guard against that, the build system generates a rule that verifies
uses-libraries: checks the metadata in the build files against the contents of a
manifest. The manifest can be available as a source file, or as part of a
prebuilt APK.

The check is implemented in [soong/scripts/manifest_check.py].

It is possible to turn off the check globally for a product by setting
`PRODUCT_BROKEN_VERIFY_USES_LIBRARIES := true` in a product makefile, or for a
particular build by setting `RELAX_USES_LIBRARY_CHECK=true`.

### Compatibility libraries - construct_context.py

Compatibility libraries are libraries that didn’t exist prior to a certain SDK
version (say, `N`), but classes in them were in the bootclasspath jars, etc.,
and in version `N` they have been separated into a standalone uses-library.
Compatibility libraries should only be in the CLC of an app if its
`targetSdkVersion` in the manifest is less than `N`.

Currently compatibility libraries only affect apps (but not other libraries).

The build system cannot see `targetSdkVersion` of an app at the time it
generates dexpreopt build rules, so it doesn't know whether to add compatibility
libaries to CLC or not. As a workaround, the build system includes all
compatibility libraries regardless of the app version, and appends some extra
logic to the dexpreopt rule that will extract `targetSdkVersion` from the
manifest and filter CLC based on that version during Ninja stage of the build,
immediately before executing the dexpreopt command (see the
soong/scripts/construct_context.py script).

As of the time of writing (January 2022), there are the following compatibility
libraries:

- org.apache.http.legacy (SDK 28)
- android.hidl.base-V1.0-java (SDK 29)
- android.hidl.manager-V1.0-java (SDK 29)
- android.test.base (SDK 30)
- android.test.mock (SDK 30)

### Manifest fixer

Sometimes uses-library tags are missing from the source manifest of a
library/app. This may happen for example if one of the transitive dependencies
of the library/app starts using another uses-library, and the library/app's
manifest isn't updated to include it.

Soong can compute some of the missing uses-library tags for a given library/app
automatically as SDK libraries in the transitive dependency closure of the
library/app. The closure is needed because a library/app may depend on a static
library that may in turn depend on an SDK library (possibly transitively via
another library).

Not all uses-library tags can be computed in this way, because some of the
uses-library dependencies are not SDK libraries, or they are not reachable via
transitive dependency closure. But when possible, allowing Soong to calculate
the manifest entries is less prone to errors and simplifies maintenance. For
example, consider a situation when many apps use some static library that adds a
new uses-library dependency -- all the apps will have to be updated. That is
difficult to maintain.

There is also a manifest merger, because sometimes the final manifest of an app
is merged from a few dependency manifests, so the final manifest installed on
devices contains a superset of uses-library tags of the source manifest of the
app.


[make/core/dex_preopt.mk]: https://cs.android.com/android/platform/superproject/+/master:build/make/core/dex_preopt.mk
[make/core/dex_preopt_config.mk]: https://cs.android.com/android/platform/superproject/+/master:build/make/core/dex_preopt_config.mk
[make/core/dex_preopt_config_merger.py]: https://cs.android.com/android/platform/superproject/+/master:build/make/core/dex_preopt_config_merger.py
[make/core/dex_preopt_odex_install.mk]: https://cs.android.com/android/platform/superproject/+/master:build/make/core/dex_preopt_odex_install.mk
[soong/dexpreopt]: https://cs.android.com/android/platform/superproject/+/master:build/soong/dexpreopt
[soong/dexpreopt/class_loader_context.go]: https://cs.android.com/android/platform/superproject/+/master:build/soong/dexpreopt/class_loader_context.go
[soong/dexpreopt/config.go]: https://cs.android.com/android/platform/superproject/+/master:build/soong/dexpreopt/config.go
[soong/dexpreopt/dexpreopt.go]: https://cs.android.com/android/platform/superproject/+/master:build/soong/dexpreopt/dexpreopt.go
[soong/java]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java
[soong/java/app.go:deps]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java/app.go?q=%22func%20\(u%20*usesLibrary\)%20deps%22
[soong/java/app.go:verifyUsesLibraries]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java/app.go?q=%22func%20\(u%20*usesLibrary\)%20verifyUsesLibraries%22
[soong/java/bootclasspath_fragment.go]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java/bootclasspath_fragment.go
[soong/java/dexpreopt.go]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java/dexpreopt.go
[soong/java/dexpreopt_bootjars.go]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java/dexpreopt_bootjars.go
[soong/java/dexpreopt_config.go]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java/dexpreopt_config.go
[soong/java/java.go:addCLCFromDep]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java/java.go?q=%22func%20addCLCfromDep%22
[soong/java/platform_bootclasspath.go]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java/platform_bootclasspath.go
[soong/scripts/construct_context.py]: https://cs.android.com/android/platform/superproject/+/master:build/soong/scripts/construct_context.py
[soong/scripts/manifest_check.py]: https://cs.android.com/android/platform/superproject/+/master:build/soong/scripts/manifest_check.py