blob: f605f6c3d972dc55baf1ae52a84d3b5b377e998b [file] [log] [blame] [view] [edit]
# Running AndroidX Benchmarks outside of AndroidX
[TOC]
# Running AndroidX Benchmarks outside of AndroidX
AndroidX-internal benchmarks can be a useful suite to test runtime, platform and
system performance/changes.
For this reason, this doc explains how to use these benchmarks outside of the
context of the AndroidX repository as APKs, and invoking them with `am
instrument`, as you would any standard test APK.
Note that AndroidX internal benchmarks are *not* a published suite with stable
releases - these benchmarks are updated continuously alongside the libraries
that use them.
To see more about writing and modifying benchmarks in the AndroidX repository,
and running them in Studio, see [this page](benchmarking.md).
## Benchmark types
Macrobenchmarks measure high level app interactions such as startup and
scrolling, and are preferred for measuring anything other than best case (AOT)
app performance.
Microbenchmarks measure specific function calls in a tight loop, and are only
designed to measure best-case, single-threaded performace for hot code.
For more info about micro vs macro, see
[here](https://developer.android.com/topic/performance/benchmarking/benchmarking-overview).
## Getting the APKs
### Android Build Page
Build server built APKs are available in the
[aosp-androidx-main branch](https://ci.android.com/builds/branches/aosp-androidx-main/grid?legacy=1).
This corresponds to builds from source [here](https://cs.android.com/androidx).
1. Click the build square under the `android_device_tests` target column (hover
over columns to see it)
2. Click 'Artifacts' at the bottom
3. Download `androidTest.zip`
Inside that (**Note: ~4.5GB**) zip file, you'll see roughly two dozen benchmark
APKs. Each of these will have a name containing the substring
`"benchmark-releaseAndroidTest.apk"`
These generally correspond to library groups and subgroups within AndroidX.
An example build artifacts page with an `androidTest.zip` file is
[here](https://ci.android.com/builds/submitted/14809214/androidx_device_tests/latest)
(buildId `14809214`)
### Local Build
You can locally build AndroidX modules with Gradle commands like the following:
```shell
# MACRObenchmark example - two APKs are produced
./gradlew compose:integration-tests:hero:sysui:sysui-macrobenchmark:assemble compose:integration-tests:hero:sysui:sysui-macrobenchmark-target:assemble
# microbenchmark example - one APK is produced
./gradlew compose:runtime:runtime:benchmark:assembleAndroidTest
```
You can find all of the available benchmark modules in
[settings.gradle](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:settings.gradle).
Each can be built with `<modulepath>:assemble`.
Output APKs can be found in `../../out/androidx`, for example:
```shell
# above MACRObenchmark example:
../../out/androidx/compose/integration-tests/hero/sysui/sysui-macrobenchmark/build/outputs/apk/release/sysui-macrobenchmark-release.apk
../../out/androidx/compose/integration-tests/hero/sysui/sysui-macrobenchmark-target/build/outputs/apk/release/sysui-macrobenchmark-target-release.apk
# above microbenchmark example:
../../out/androidx/compose/runtime/runtime/benchmark/build/outputs/apk/androidTest/release/benchmark-release-androidTest.apk
```
For more information, see
[Benchmarking in CI](https://developer.android.com/topic/performance/benchmarking/benchmarking-in-ci)
## MACRObenchmark {#macrobenchmark}
Macrobenchmarks require two apks, but are otherwise simpler to setup and run:
* Clocks are unlocked (partly because this affects deadlines like frame jank,
and partly because they're expected to have non-CPU work)
* Compilation (verify, speed-profile, speed) is configured and performed by
the benchmark library itself
Note while profiling is
[partly supported](https://developer.android.com/topic/performance/benchmarking/macrobenchmark-instrumentation-args#profiling-mode),
it is not as comprehensive as in microbenchmark, due to the multi-process nature
of macrobenchmarks.
Many benchmarks can be parameterized across
[CompilationMode](https://developer.android.com/reference/kotlin/androidx/benchmark/macro/CompilationMode)
(an abstraction over art compilation filters / OS variations), or
[StartupMode](https://developer.android.com/reference/androidx/benchmark/macro/StartupMode)
(for startup benchmarks). For startup benchmarks, COLD startups are the ones
worth running / measuring.
Macrobenchmarks always output (and derive timing from) Perfetto traces, but can
be configured to capture method traces - note that it's intrusive to measure, as
it's not run in a separate phase. Same argument is used, as below in
microbenchmarks:
```shell
-e androidx.benchmark.profiling.mode MethodTracing
```
### Running MACRObenchmarks
MACRObenchmark tests are always installed in apk pairs, one for each of the test
and target packages:
```shell
adb install -r <modulename>-macrobenchmark-release.apk
adb install -r <modulename>-macrobenchmark-target_for_<modulename>-macrobenchmark-release.apk
```
Individual tests can always be run with commands of the format:
```shell
adb shell am instrument -e "class" "<androidx.somepackage.BenchmarkClass>#<benchmarkMethod>" -w <BenchmarkPackage>/androidx.test.runner.AndroidJUnitRunner
```
You can discover the classes contained within from
[code search](https://cs.android.com/androidx), or find available classes with a
shell command:
```shell
adb shell am instrument -e log true -w <BenchmarkPackage>/androidx.test.runner.AndroidJUnitRunner
```
See examples below, note parameterization is often required, especially for
macrobenchmarks.
#### Pokedex Hero Macrobenchmarks
Install
```
adb install -r compose-integration-tests-hero-pokedex-pokedex-macrobenchmark-release.apk
adb install -r compose-integration-tests-hero-pokedex-pokedex-macrobenchmark-target_for_compose-integration-tests-hero-pokedex-pokedex-macrobenchmark-release.apk
```
Run \
Note: parameterization of compilation mode is limited, to limit CI runtimes
```shell
adb shell am instrument -e "class" "androidx.compose.integration.hero.pokedex.macrobenchmark.PokedexScrollBenchmark#scrollHomeCompose[compilation=Full,eSTS=true,eSET=true]" -w androidx.compose.integration.hero.pokedex.macrobenchmark/androidx.test.runner.AndroidJUnitRunner
```
#### Jetsnack Hero Macrobenchmarks
[JetsnackScrollBenchmark](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:compose/integration-tests/hero/jetsnack/jetsnack-macrobenchmark/src/main/java/androidx/compose/integration/hero/jetsnack/macrobenchmark/JetsnackScrollBenchmark.kt)
and
[JetsnackStartupBenchmark](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:compose/integration-tests/hero/jetsnack/jetsnack-macrobenchmark/src/main/java/androidx/compose/integration/hero/jetsnack/macrobenchmark/JetsnackStartupBenchmark.kt)
are higher level, more realistic scroll/startup benchmarks that are also
important.
Install:
```shell
adb install -r compose-integration-tests-hero-jetsnack-jetsnack-macrobenchmark-release.apk
adb install -r compose-integration-tests-hero-jetsnack-jetsnack-macrobenchmark-target_for_compose-integration-tests-hero-jetsnack-jetsnack-macrobenchmark-release.apk
```
Run:
```shell
adb shell am instrument -e "class" "androidx.compose.integration.hero.jetsnack.macrobenchmark.JetsnackScrollBenchmark#scrollHome[compilation=BaselineProfile]" -w androidx.compose.integration.hero.jetsnack.macrobenchmark/androidx.test.runner.AndroidJUnitRunner
```
#### Compose Minimal Macrobenchmarks
[TrivialListScrollBenchmark](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:compose/integration-tests/macrobenchmark/src/main/java/androidx/compose/integration/macrobenchmark/TrivialListScrollBenchmark.kt?q=compose%2Fintegration-tests%2Fmacrobenchmark%2Fsrc%2Fmain%2Fjava%2Fandroidx%2Fcompose%2Fintegration%2Fmacrobenchmark%2FTrivialListScrollBenchmark.kt)
and
[SmallListStartupBenchmark](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:compose/integration-tests/macrobenchmark/src/main/java/androidx/compose/integration/macrobenchmark/SmallListStartupBenchmark.kt)
are good low-level starting points, since they cover the most critical building
blocks of UI display and scrolling.
Install:
```shell
adb install -r compose-integration-tests-macrobenchmark-release.apk
adb install -r compose-integration-tests-macrobenchmark-target_for_compose-integration-tests-macrobenchmark-release.apk
```
Run:
```shell
adb shell am instrument -e "class" "androidx.compose.integration.macrobenchmark.TrivialListScrollBenchmark#start[compilation=BaselineProfile]" -w androidx.compose.integration.macrobenchmark/androidx.test.runner.AndroidJUnitRunner
```
## Microbenchmark
Microbenchmarks measure code entrypoints, and are somewhat more stable than
[macrobenchmarks (see below)](#macrobenchmark), but they do not yet support
being built with R8. Support is being added to the gradle build system
currently. See
[this doc](https://developer.android.com/topic/performance/benchmarking/benchmarking-overview)
for more info about micro vs macro.
### Running the Microbenchmark
Install the benchmark, e.g.: `adb install
compose-foundation-foundation-benchmarks.apk` Then prepare the device by locking
clocks and disabling JIT (requires adb root):
```shell
benchmark/gradle-plugin/src/main/resources/scripts/disableJit.sh
benchmark/gradle-plugin/src/main/resources/scripts/lockClocks.sh
```
These script are available
[here](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:benchmark/gradle-plugin/src/main/resources/scripts),
and can be run from host or device.
Then speed compile the app, to ensure stable workload:
```shell
cmd package compile -f -m speed androidx.compose.foundation.benchmark.test
```
Note: this command as well as all general parameters can be found in the xml
file alongside the benchmark apk.
Now you can invoke a single test from the command line:
```shell
adb shell am instrument -e "class" "androidx.compose.foundation.benchmark.lazy.LazyListScrollingBenchmark\#scrollProgrammatically\_newItemComposed\[LazyColumn\]" -w androidx.compose.foundation.benchmark.test/androidx.benchmark.junit4.AndroidBenchmarkRunner
```
> NOTE: Device screen must be on, so activities can be launched to host UI.
>
> Locally, use Developer Options \> Stay Awake.
>
> If you're running with Tradefed, this can be configured for your suite:
>
> ```xml
> <lab_preparer class="com.google.android.tradefed.targetprep.GoogleDeviceSetup">
> <option name="disable" value="true" />
> <option name="screen-always-on" value="on" />
> <option name="screen-adaptive-brightness" value="off" />
> </lab_preparer>
> ```
This will have a very basic output on the command line, but CLI text output
doesn't include Perfetto trace (captured by default), method trace (recommended,
captured in separate phase), or json output with detailed metrics, these need to
be pulled separately. See
[Benchmarking in Continuous Integration](https://developer.android.com/topic/performance/benchmarking/benchmarking-in-ci)
for more info on pulling trace/json output.
You can also modify instrumentation args to e.g. add profiling, like the
following which captures a method trace after timing measurements:
```shell
adb shell am instrument -e "class"
"androidx.compose.foundation.benchmark.lazy.LazyListScrollingBenchmark\#scrollProgrammatically\_newItemComposed\[LazyColumn\]"
-e androidx.benchmark.profiling.mode MethodTracing -w
androidx.compose.foundation.benchmark.test/androidx.benchmark.junit4.AndroidBenchmarkRunner
```
See the
[public docs for instrumentation args](https://developer.android.com/topic/performance/benchmarking/microbenchmark-instrumentation-args)
to see what can be configured.
### Where to start:
[LazyListScrollingBenchmark](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:compose/foundation/foundation/benchmark/src/androidTest/java/androidx/compose/foundation/benchmark/lazy/LazyListScrollingBenchmark.kt)
and
[scrollProgrammattically_newItemComposed](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:compose/foundation/foundation/benchmark/src/androidTest/java/androidx/compose/foundation/benchmark/lazy/LazyListScrollingBenchmark.kt;l=81;drc=e6d33dd5d0a60001a5784d84123b05308d35f410)
in particular is a good place to start, as it captures much of the cost of the
scrolling container in Compose (albeit with a trivial layout).
[PokedexScrollingBenchmark](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:compose/integration-tests/hero/pokedex/pokedex-macrobenchmark/src/main/java/androidx/compose/integration/hero/pokedex/macrobenchmark/PokedexScrollBenchmark.kt)
is a higher level benchmark, and also a good place to start.
You can also pick a specific class to run - note that the entire suite will take
10s of minutes to run, and heavily parameterized classes can take many minutes
as well.