| # Bionic Benchmarks |
| |
| [TOC] |
| |
| ## libc benchmarks (bionic-benchmarks) |
| |
| `bionic-benchmarks` is a command line tool for measuring the runtimes of libc functions. It is built |
| on top of [Google Benchmark](https://github.com/google/benchmark) with some additions to organize |
| tests into suites. |
| |
| ### Device benchmarks |
| |
| $ mmma bionic/benchmarks |
| $ adb root |
| $ adb sync data |
| $ adb shell /data/benchmarktest/bionic-benchmarks/bionic-benchmarks |
| $ adb shell /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks |
| |
| By default, `bionic-benchmarks` runs all of the benchmarks in alphabetical order. Pass |
| `--benchmark_filter=getpid` to run just the benchmarks with "getpid" in their name. |
| |
| Note that we also build _static_ benchmark binaries. |
| They're useful for testing on devices running different versions of Android, or running non-Android OSes. |
| Those binaries are called `bionic-benchmarks-static` instead. |
| Copy from `out/target/product/<device>/symbols/data/benchmarktest64/bionic-benchmarks-static` instead of |
| `out/target/product/<device>/data/benchmarktest64/bionic-benchmarks-static` if you want symbols for perf(1). |
| |
| ### Host benchmarks |
| |
| See the `benchmarks/run-on-host.sh` script. The host benchmarks can be run with 32-bit or 64-bit |
| Bionic, or the host glibc. |
| |
| ### XML suites |
| |
| Suites are stored in the `suites/` directory and can be chosen with the command line flag |
| `--bionic_xml`. |
| |
| To choose a specific XML file, use the `--bionic_xml=FILE.XML` option. By default, this option |
| searches for the XML file in the `suites/` directory. If it doesn't exist in that directory, then |
| the file will be found as relative to the current directory. If the option specifies the full path |
| to an XML file such as `/data/nativetest/suites/example.xml`, it will be used as-is. |
| |
| If no XML file is specified through the command-line option, the default is to use `suites/full.xml`. |
| However, for the host bionic benchmarks (`bionic-benchmarks-glibc`), the default is to use |
| `suites/host.xml`. |
| |
| ### XML suite format |
| |
| The format for a benchmark is: |
| |
| ``` |
| <fn> |
| <name>BM_sample_benchmark</name> |
| <cpu><optional_cpu_to_lock></cpu> |
| <iterations><optional_iterations_to_run></iterations> |
| <args><space separated list of function args|shorthand></args> |
| </fn> |
| ``` |
| |
| XML-specified values for iterations and cpu take precedence over those specified via command line |
| (via `--bionic_iterations` and `--bionic_cpu`, respectively.) |
| |
| To make small changes in runs, you can also schedule benchmarks by passing in their name and a |
| space-separated list of arguments via the `--bionic_extra` command line flag, e.g. |
| `--bionic_extra="BM_string_memcpy AT_COMMON_SIZES"` or `--bionic_extra="BM_string_memcmp 32 8 8"` |
| |
| Note that benchmarks will run normally if extra arguments are passed in, and it will fail |
| with a segfault if too few are passed in. |
| |
| ### Shorthand |
| |
| For the sake of brevity, multiple runs can be scheduled in one XML element by putting one of the |
| following in the args field: |
| |
| NUM_PROPS |
| MATH_COMMON |
| AT_ALIGNED_<ONE|TWO>BUF |
| AT_<any power of two between 2 and 16384>_ALIGNED_<ONE|TWO>BUF |
| AT_COMMON_SIZES |
| |
| Definitions for these can be found in bionic_benchmarks.cpp, and example usages can be found in |
| the suites directory. |
| |
| ### Unit Tests |
| |
| `bionic-benchmarks` also has its own set of unit tests, which can be run from the binary in |
| `/data/nativetest[64]/bionic-benchmarks-tests` |
| |
| ## Process startup time (bionic-spawn-benchmarks) |
| |
| The `spawn/` subdirectory has a few benchmarks measuring the time used to start simple programs |
| (e.g. Toybox's `true` and `sh -c true`). Run it on a device like so: |
| |
| m bionic-spawn-benchmarks |
| adb root |
| adb sync data |
| adb shell /data/benchmarktest/bionic-spawn-benchmarks/bionic-spawn-benchmarks |
| adb shell /data/benchmarktest64/bionic-spawn-benchmarks/bionic-spawn-benchmarks |
| |
| Google Benchmark reports both a real-time figure ("Time") and a CPU usage figure. For these |
| benchmarks, the CPU measurement only counts time spent in the thread calling `posix_spawn`, not that |
| spent in the spawned process. The real-time is probably more useful, and it is the figure used to |
| determine the iteration count. |
| |
| Locking the CPU frequency seems to improve the results of these benchmarks significantly, and it |
| reduces variability. |
| |
| ## Google Benchmark notes |
| |
| ### Repetitions |
| |
| Google Benchmark uses two settings to control how many times to run each benchmark, "iterations" and |
| "repetitions". By default, the repetition count is one. Google Benchmark runs the benchmark a few |
| times to determine a sufficiently-large iteration count. |
| |
| Google Benchmark can optionally run a benchmark run repeatedly and report statistics (median, mean, |
| standard deviation) for the runs. To do so, pass the `--benchmark_repetitions` option, e.g.: |
| |
| # ./bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --benchmark_repetitions=4 |
| ... |
| ------------------------------------------------------------------- |
| Benchmark Time CPU Iterations |
| ------------------------------------------------------------------- |
| BM_stdlib_strtoll 27.7 ns 27.7 ns 25290525 |
| BM_stdlib_strtoll 27.7 ns 27.7 ns 25290525 |
| BM_stdlib_strtoll 27.7 ns 27.7 ns 25290525 |
| BM_stdlib_strtoll 27.8 ns 27.7 ns 25290525 |
| BM_stdlib_strtoll_mean 27.7 ns 27.7 ns 4 |
| BM_stdlib_strtoll_median 27.7 ns 27.7 ns 4 |
| BM_stdlib_strtoll_stddev 0.023 ns 0.023 ns 4 |
| |
| There are 4 runs, each with 25290525 iterations. Measurements for the individual runs can be |
| suppressed if they aren't needed: |
| |
| # ./bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --benchmark_repetitions=4 --benchmark_report_aggregates_only |
| ... |
| ------------------------------------------------------------------- |
| Benchmark Time CPU Iterations |
| ------------------------------------------------------------------- |
| BM_stdlib_strtoll_mean 27.8 ns 27.7 ns 4 |
| BM_stdlib_strtoll_median 27.7 ns 27.7 ns 4 |
| BM_stdlib_strtoll_stddev 0.043 ns 0.043 ns 4 |
| |
| ### CPU frequencies |
| |
| To get consistent results between runs, it can sometimes be helpful to restrict a benchmark to |
| specific cores, or to lock cores at specific frequencies. Some phones have a big.LITTLE core setup, |
| or at least allow some cores to run at higher frequencies than others. |
| |
| A core can be selected for `bionic-benchmarks` using the `--bionic_cpu` option or using the |
| `taskset` utility. e.g. A Pixel 3 device has 4 Kryo 385 Silver cores followed by 4 Gold cores: |
| |
| blueline:/ # /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --bionic_cpu=0 |
| ... |
| ------------------------------------------------------------ |
| Benchmark Time CPU Iterations |
| ------------------------------------------------------------ |
| BM_stdlib_strtoll 64.2 ns 63.6 ns 11017493 |
| |
| blueline:/ # /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --bionic_cpu=4 |
| ... |
| ------------------------------------------------------------ |
| Benchmark Time CPU Iterations |
| ------------------------------------------------------------ |
| BM_stdlib_strtoll 21.8 ns 21.7 ns 33167103 |
| |
| A similar result can be achieved using `taskset`. The first parameter is a bitmask of core numbers |
| to pass to `sched_setaffinity`: |
| |
| blueline:/ # taskset f /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll |
| ... |
| ------------------------------------------------------------ |
| Benchmark Time CPU Iterations |
| ------------------------------------------------------------ |
| BM_stdlib_strtoll 64.3 ns 63.6 ns 10998697 |
| |
| blueline:/ # taskset f0 /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll |
| ... |
| ------------------------------------------------------------ |
| Benchmark Time CPU Iterations |
| ------------------------------------------------------------ |
| BM_stdlib_strtoll 21.3 ns 21.2 ns 33094801 |
| |
| To lock the CPU frequency, use the sysfs interface at `/sys/devices/system/cpu/cpu*/cpufreq/`. |
| Changing the scaling governor to `performance` suppresses the warning that Google Benchmark |
| otherwise prints: |
| |
| ***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead. |
| |
| Some devices have a `perf-setup.sh` script that locks CPU and GPU frequencies. Some TradeFed |
| benchmarks appear to be using the script. For more information: |
| * run `adb shell perf-setup.sh` to execute the script, it is already by default be installed on device for eng and userdebug build |
| * see: https://android.googlesource.com/platform/platform_testing/+/refs/heads/main/scripts/perf-setup/ |
