Running AndroidX Benchmarks outside of AndroidX

AndroidX-internal benchmarks can be a useful suite to test runtime, platform and system performance/changes.

For this reason, this doc explains how to use these benchmarks outside of the context of the AndroidX repository as APKs, and invoking them with am instrument, as you would any standard test APK.

Note that AndroidX internal benchmarks are not a published suite with stable releases - these benchmarks are updated continuously alongside the libraries that use them.

To see more about writing and modifying benchmarks in the AndroidX repository, and running them in Studio, see this page.

Benchmark types

Macrobenchmarks measure high level app interactions such as startup and scrolling, and are preferred for measuring anything other than best case (AOT) app performance.

Microbenchmarks measure specific function calls in a tight loop, and are only designed to measure best-case, single-threaded performace for hot code.

For more info about micro vs macro, see here.

Getting the APKs

Android Build Page

Build server built APKs are available in the aosp-androidx-main branch. This corresponds to builds from source here.

Click the build square under the android_device_tests target column (hover over columns to see it)
Click ‘Artifacts’ at the bottom
Download androidTest.zip

Inside that (Note: ~4.5GB) zip file, you'll see roughly two dozen benchmark APKs. Each of these will have a name containing the substring "benchmark-releaseAndroidTest.apk"

These generally correspond to library groups and subgroups within AndroidX.

An example build artifacts page with an androidTest.zip file is here (buildId 14809214)

Local Build

You can locally build AndroidX modules with Gradle commands like the following:

# MACRObenchmark example - two APKs are produced
./gradlew compose:integration-tests:hero:sysui:sysui-macrobenchmark:assemble compose:integration-tests:hero:sysui:sysui-macrobenchmark-target:assemble

# microbenchmark example - one APK is produced
./gradlew compose:runtime:runtime:benchmark:assembleAndroidTest

You can find all of the available benchmark modules in settings.gradle. Each can be built with <modulepath>:assemble.

Output APKs can be found in ../../out/androidx, for example:

# above MACRObenchmark example:
../../out/androidx/compose/integration-tests/hero/sysui/sysui-macrobenchmark/build/outputs/apk/release/sysui-macrobenchmark-release.apk
../../out/androidx/compose/integration-tests/hero/sysui/sysui-macrobenchmark-target/build/outputs/apk/release/sysui-macrobenchmark-target-release.apk

# above microbenchmark example:
../../out/androidx/compose/runtime/runtime/benchmark/build/outputs/apk/androidTest/release/benchmark-release-androidTest.apk

For more information, see Benchmarking in CI

MACRObenchmark

Macrobenchmarks require two apks, but are otherwise simpler to setup and run:

Clocks are unlocked (partly because this affects deadlines like frame jank, and partly because they're expected to have non-CPU work)
Compilation (verify, speed-profile, speed) is configured and performed by the benchmark library itself

Note while profiling is partly supported, it is not as comprehensive as in microbenchmark, due to the multi-process nature of macrobenchmarks.

Many benchmarks can be parameterized across CompilationMode (an abstraction over art compilation filters / OS variations), or StartupMode (for startup benchmarks). For startup benchmarks, COLD startups are the ones worth running / measuring.

Macrobenchmarks always output (and derive timing from) Perfetto traces, but can be configured to capture method traces - note that it‘s intrusive to measure, as it’s not run in a separate phase. Same argument is used, as below in microbenchmarks:

-e androidx.benchmark.profiling.mode MethodTracing

Running MACRObenchmarks

MACRObenchmark tests are always installed in apk pairs, one for each of the test and target packages:

adb install -r <modulename>-macrobenchmark-release.apk
adb install -r <modulename>-macrobenchmark-target_for_<modulename>-macrobenchmark-release.apk

Individual tests can always be run with commands of the format:

adb shell am instrument -e "class" "<androidx.somepackage.BenchmarkClass>#<benchmarkMethod>" -w <BenchmarkPackage>/androidx.test.runner.AndroidJUnitRunner

You can discover the classes contained within from code search, or find available classes with a shell command:

adb shell am instrument -e log true -w <BenchmarkPackage>/androidx.test.runner.AndroidJUnitRunner

See examples below, note parameterization is often required, especially for macrobenchmarks.

Pokedex Hero Macrobenchmarks

Install

adb install -r compose-integration-tests-hero-pokedex-pokedex-macrobenchmark-release.apk
adb install -r compose-integration-tests-hero-pokedex-pokedex-macrobenchmark-target_for_compose-integration-tests-hero-pokedex-pokedex-macrobenchmark-release.apk

Run
Note: parameterization of compilation mode is limited, to limit CI runtimes

adb shell am instrument -e "class" "androidx.compose.integration.hero.pokedex.macrobenchmark.PokedexScrollBenchmark#scrollHomeCompose[compilation=Full,eSTS=true,eSET=true]" -w androidx.compose.integration.hero.pokedex.macrobenchmark/androidx.test.runner.AndroidJUnitRunner

Jetsnack Hero Macrobenchmarks

JetsnackScrollBenchmark and JetsnackStartupBenchmark are higher level, more realistic scroll/startup benchmarks that are also important.

Install:

adb install -r compose-integration-tests-hero-jetsnack-jetsnack-macrobenchmark-release.apk
adb install -r compose-integration-tests-hero-jetsnack-jetsnack-macrobenchmark-target_for_compose-integration-tests-hero-jetsnack-jetsnack-macrobenchmark-release.apk

Run:

adb shell am instrument -e "class" "androidx.compose.integration.hero.jetsnack.macrobenchmark.JetsnackScrollBenchmark#scrollHome[compilation=BaselineProfile]" -w androidx.compose.integration.hero.jetsnack.macrobenchmark/androidx.test.runner.AndroidJUnitRunner

Compose Minimal Macrobenchmarks

TrivialListScrollBenchmark and SmallListStartupBenchmark are good low-level starting points, since they cover the most critical building blocks of UI display and scrolling.

Install:

adb install -r compose-integration-tests-macrobenchmark-release.apk
adb install -r compose-integration-tests-macrobenchmark-target_for_compose-integration-tests-macrobenchmark-release.apk

Run:

adb shell am instrument -e "class" "androidx.compose.integration.macrobenchmark.TrivialListScrollBenchmark#start[compilation=BaselineProfile]" -w androidx.compose.integration.macrobenchmark/androidx.test.runner.AndroidJUnitRunner

Microbenchmark

Microbenchmarks measure code entrypoints, and are somewhat more stable than macrobenchmarks (see below), but they do not yet support being built with R8. Support is being added to the gradle build system currently. See this doc for more info about micro vs macro.

Running the Microbenchmark

Install the benchmark, e.g.: adb install compose-foundation-foundation-benchmarks.apk Then prepare the device by locking clocks and disabling JIT (requires adb root):

benchmark/gradle-plugin/src/main/resources/scripts/disableJit.sh
benchmark/gradle-plugin/src/main/resources/scripts/lockClocks.sh

These script are available here, and can be run from host or device.

Then speed compile the app, to ensure stable workload:

cmd package compile -f -m speed androidx.compose.foundation.benchmark.test

Note: this command as well as all general parameters can be found in the xml file alongside the benchmark apk.

Now you can invoke a single test from the command line:

adb shell am instrument -e "class" "androidx.compose.foundation.benchmark.lazy.LazyListScrollingBenchmark\#scrollProgrammatically\_newItemComposed\[LazyColumn\]" -w androidx.compose.foundation.benchmark.test/androidx.benchmark.junit4.AndroidBenchmarkRunner

NOTE: Device screen must be on, so activities can be launched to host UI.
Locally, use Developer Options > Stay Awake.
If you're running with Tradefed, this can be configured for your suite:
    <lab_preparer class="com.google.android.tradefed.targetprep.GoogleDeviceSetup">
        <option name="disable" value="true" />
        <option name="screen-always-on" value="on" />
        <option name="screen-adaptive-brightness" value="off" />
    </lab_preparer>

This will have a very basic output on the command line, but CLI text output doesn't include Perfetto trace (captured by default), method trace (recommended, captured in separate phase), or json output with detailed metrics, these need to be pulled separately. See Benchmarking in Continuous Integration for more info on pulling trace/json output.

You can also modify instrumentation args to e.g. add profiling, like the following which captures a method trace after timing measurements:

adb shell am instrument -e "class"
"androidx.compose.foundation.benchmark.lazy.LazyListScrollingBenchmark\#scrollProgrammatically\_newItemComposed\[LazyColumn\]"
-e androidx.benchmark.profiling.mode MethodTracing -w
androidx.compose.foundation.benchmark.test/androidx.benchmark.junit4.AndroidBenchmarkRunner

See the public docs for instrumentation args to see what can be configured.

Where to start:

LazyListScrollingBenchmark and scrollProgrammattically_newItemComposed in particular is a good place to start, as it captures much of the cost of the scrolling container in Compose (albeit with a trivial layout).

PokedexScrollingBenchmark is a higher level benchmark, and also a good place to start.

You can also pick a specific class to run - note that the entire suite will take 10s of minutes to run, and heavily parameterized classes can take many minutes as well.