Jakub Gielzak | 206ba4c | 2024-01-04 13:49:41 +0000 | [diff] [blame] | 1 | # Overview |
| 2 | |
| 3 | The project provides an easy way to save before/after CPU traces from Microbenchmark runs, and compare them visually using Differential Flame Graphs. |
| 4 | |
| 5 |  |
| 6 | |
| 7 | Areas where the code got slower are highlighted in red, while areas that are now faster are marked in blue; the intensity of the colour is proportional to the size of the difference. |
| 8 | |
| 9 | See also the [end-to-end demo (video)](https://drive.google.com/file/d/119nI_zlAMbTHzh-Rdzf8UuUVCGEKnKFQ/view?usp=drive_link&resourcekey=0-SRRmKgVZYfAlnkL4Hvh-cg). |
| 10 | |
| 11 | # Usage |
| 12 | |
| 13 | ## Interacting with the script |
| 14 | |
| 15 | - Overview of all commands: `./bench-flame-diff.sh -h` |
| 16 | - Help for a specific command: `./bench-flame-diff.sh <command> -h` |
| 17 | |
| 18 | ## First usage |
| 19 | |
| 20 | On first usage, initialise all dependencies by running: `./bench-flame-diff.sh init` |
| 21 | |
| 22 | ## General workflow |
| 23 | |
| 24 | 1. Run a specific Microbenchmark with CPU Stack sampling enabled (see below for instructions) |
| 25 | 1. Save the trace as _base_ for comparison using `./bench-flame-diff.sh save`. It's worth picking a good names for the saved traces since you're likely going to e.g. re-use the _base_ while iterating on code changes. |
| 26 | 1. Apply changes in your code and run the same benchmark as in step 1 |
| 27 | 1. Save the trace as _current_ `./bench-flame-diff.sh save` |
| 28 | 1. Compare both traces using `./bench-flame-diff.sh diff` which will create and open a diff in a web browser |
| 29 | 1. Toggle between graphs using the buttons on the top: |
| 30 | - `base`: flamegraph for the _base_ trace |
| 31 | - `base-vs-curr`: differential flame graph showing _base_ vs _current_ on the _base_ trace |
| 32 | - `curr`: flamegraph for the _current_ trace |
| 33 | - `curr-vs-base`: differential flame graph showing _base_ vs _current_ on the _current_ trace |
| 34 | 1. You can later go back to generated diffs using `./bench-flame-diff.sh open` |
| 35 | |
| 36 | # Misc |
| 37 | |
| 38 | ## Enabling stack sampling in Benchmark traces |
| 39 | |
| 40 | This can be done in CLI or by editing `build.gradle`. Full documentation is [here](https://developer.android.com/topic/performance/benchmarking/microbenchmark-profile). |
| 41 | |
| 42 | Quick CLI example: |
| 43 | ``` |
| 44 | # pick a target benchmark |
| 45 | tgt=:compose:foundation:foundation-benchmark:connectedCheck |
| 46 | |
| 47 | # create a regex that targets a specific benchmark (test) |
| 48 | test_rx="androidx.compose.foundation.benchmark.lazy.LazyListScrollingBenchmark.scrollProgrammatically_noNewItems\[.*Row.*\]" |
| 49 | |
| 50 | # run the benchmark and gather a 5 second (default) stack sample at 1000 Hz (default) |
| 51 | ./gradlew $tgt -Pandroid.testInstrumentationRunnerArguments.tests_regex="$test_rx" \ |
| 52 | -P android.testInstrumentationRunnerArguments.androidx.benchmark.profiling.mode=StackSampling \ |
| 53 | -P android.testInstrumentationRunnerArguments.androidx.benchmark.profiling.sampleDurationSeconds=5 \ |
| 54 | -P android.testInstrumentationRunnerArguments.androidx.benchmark.profiling.sampleFrequency=1000 |
| 55 | ``` |
| 56 | |
| 57 | ## CLI completion |
| 58 | |
Chris Craik | 2c24668 | 2024-03-18 10:42:28 -0700 | [diff] [blame] | 59 | Generate shell-specific completion files with `./generate-completion.sh`. |
| 60 | |
| 61 | Then, source in your shell config, e.g.: |
Jakub Gielzak | 206ba4c | 2024-01-04 13:49:41 +0000 | [diff] [blame] | 62 | - For `bash`: `dst="$(pwd)/completion_bash.sh"; echo "source '$dst'" >> ~/.bashrc` |
| 63 | - For `zsh`: `dst="$(pwd)/completion_zsh.sh"; echo "source '$dst'" >> ~/.zshrc` |
| 64 | |
| 65 | After restarting the shell session, you will be able to 'tab-autocomplete' commands and argument names. |
| 66 | |
| 67 | # Dependencies |
| 68 | |
| 69 | On top of dependencies discoverable with `./gradlew app:dependencies` the project depends on: |
| 70 | - https://github.com/brendangregg/FlameGraph |
| 71 | - https://android.googlesource.com/platform/system/extras/+/refs/heads/main/simpleperf/scripts |
| 72 | |
| 73 | Both are fetched from the network in the `init` command and pinned to known-good-revisions. |
| 74 | |
| 75 | # Reporting issues |
| 76 | |
| 77 | File an issue on Buganizer using [this link](https://b.corp.google.com/issues/new?component=1229612&hotlistIds=3622386&hotlistIds=5709693&assignee=jgielzak@google.com&title=bench-flame-diff:%20) or reach out directly to [jgielzak@](http://go/moma/chat?with=jgielzak). |
| 78 | |
| 79 | Known issues and future work items are tracked [here](https://b.corp.google.com/hotlists/5709693). |