docs/Testing.md

# Testing the NDK

The latest version of this document is available at
https://android.googlesource.com/platform/ndk/+/master/docs/Testing.md.

The NDK tests are built as part of a normal build (with `checkbuild.py`) and run
with `run_tests.py`. See [Building.md] for more instructions on building the
NDK.

From the NDK source directory (`./ndk` within the directory you ran `repo init`
in, or the root of the cloned directory if you cloned only the NDK project).

```bash
$ ./checkbuild.py  # Build the NDK and tests.
$ ./run_tests.py
```

Running the tests requires `adb` in your path and compatible devices attached.
If you're having trouble with the version from the SDK manager, try a version
built fresh from AOSP.

The test runner will look for any attached devices that match the
requirements listed in the `devices` section of the test configuration file (see
[qa\_config.json] for the defaults, or use `--config` to choose your own). Each
test will be run on all devices compatible with that test.

The full QA configuration takes roughly 10 minutes to run (P920 Linux host, 4
Galaxy Nexūs for Jelly Bean, 2 Pixels for Pie, 1 emulator for x86-64 Pie).
Attaching multiple devices will allow the test runner to shard tests among those
devices. Pushing tests can take up a large portion of the test time (3 of the 10
minutes quoted above), but `adb push --sync` is used to prevent unnecessary
pushes on reruns or when only a subset of tests are rebuilt.

The tests can be rebuilt without running `checkbuild.py` (which is necessary in
the case of not having a full NDK checkout, as you might when running the
Windows tests on a release from the build server) with `run_tests.py --rebuild`.

[qa\_config.json]: ../qa_config.json
[Building.md]: Building.md

## Restricting Test Configurations

By default, all of the configurations we test are built from both
`checkbuild.py` and `run_tests.py --rebuild`. This runs tens of thousands of
test executables. Each test is built in 4 different configurations (once for
each ABI) at time of writing. The set of configurations built can be restricted
in two ways.

First, `run_tests.py --config myconfig.json` will use an alternate test
configuration file (the default is `qa_config.json`).

Second, and simpler for a development workflow, the following flag can be used
to restrict the configurations (the presence of any of this flag will override
the matching entry in the config file, but otherwise the config file is obeyed):

```bash
$ ./run_tests.py --rebuild --abi armeabi-v7a
```

Configuration filtering flags are repeatable. For example, `--abi armeabi-v7a
--abi x86` will build both armeabi-v7a and x86 tests.

Beyond restricting test configurations, the tests themselves can be filtered
with the `--filter` flag:

```bash
$ ./run_tests.py --filter test-googletest-full
```

Test filters support wildcards (as implemented by Python's `fnmatch.fnmatch`).
The filter flag may be combined with the build configuration flags.

Putting this all together, a single test can be rebuilt and run for just
armeabi-v7a, with the following command:

```bash
$ ./run_tests.py --rebuild \
    --abi armeabi-v7a \
    --filter test-googletest-full
```

## Testing Releases

When testing a release candidate, your first choice should be to run the test
artifacts built on the build server for the given build. This is the
ndk-tests.tar.bz2 artifact in the same directory as the NDK zip. Extract the
tests somewhere, and then run:

```bash
$ ./run_tests.py path/to/extracted/tests
```

For Windows, test artifacts are not available since we cross compile the NDK
from Linux rather than building on Windows. We want to make sure the Windows
binaries we build work *on* Windows (using wine would only tell us that they
work on wine, which may not be bug compatible with Windows), so those must be
built on the test machine before they can be run. To use the fetched NDK to
build the tests, run:

```bash
$ ./run_tests.py --rebuild --ndk path/to/extracted/ndk out
```

## Broken and Unsupported Tests

To mark tests as currently broken or as unsupported for a given configuration,
add a `test_config.py` to the test's root directory (in the same directory as
`jni/`).

Unsupported tests will not be built or run.

Broken tests will be built and run, and the result of the test will be inverted.
A test that fails will become an "EXPECTED FAILURE" and not be counted as a
failure, whereas a passing test will become an "UNEXPECTED SUCCESS" and count as
a failure.

By default, `run_tests.py` will hide expected failures from the output since the
caller is most likely only interested in seeing what effect their change had. To
see the list of expected failures, pass `--show-all`.

Here's an example `test_config.py` that marks this test as broken when building
for arm64 and unsupported when running on a pre-Lollipop device:

```python
from typing import Optional

from ndk.test.devices import Device
from ndk.test.types import Test


def build_broken(test: Test) -> tuple[Optional[str], Optional[str]]:
    if test.abi == 'arm64-v8a':
        return test.abi, 'https://github.com/android-ndk/ndk/issues/foo'
    return None, None


def run_unsupported(test: Test, device: Device) -> Optional[str]:
    if device.version < 21:
        return f'{device.version}'
    return None
```

The `*_broken` checks return a tuple of `(broken_configuration, bug_url)` if the
given configuration is known to be broken, else `(None, None)`.

The `*_unsupported` checks return `broken_configuration` if the given
configuration is unsupported, else `None`.

The configuration is specified by the following arguments:

* `abi`: The ABI being built for.
* `platform`: The platform version being *built* for. Not necessarily the
  platform version that the test will be run on.
* `device_api`: The API level of the device the test will be run on.
* `name`: This is the name of the test executable being run. For libc++ tests
  built by LIT, the executable will be `foo.pass.cpp.exe`, but `name` will be
  `foo.pass`.


## Devices and Emulators

For testing a release, make sure you're testing against the released user builds
of Android.

For Nexus/Pixel devices, factory images are available here:
https://developers.google.com/android/nexus/images.

For emulators, use emulator images from the SDK rather than from a platform
build, as these are what our users will be using. Note that some NDK tests
(namely test-googletest-full and asan-smoke) are known to break between emulator
updates. It is not known whether these are NDK bugs, emulator bugs, or x86_64
system image bugs.

After installing the emulator images from the SDK manager, they can be
configured and launched for testing with (assuming the SDK tools directory is in
your path):

```bash
$ android create avd --name $NAME --target android-$LEVEL --abi $ABI
$ emulator -avd $NAME
```

This will create and launch a new virtual device.

ARM64 emulators are available in the SDK manager but emulated testing (as
compared to virtualized testing on x86_64) is very slow. Physical devices are
recommended for testing ARM64.

## Windows VMs

Warning: the process below hasn't been tested in a very long time. Googlers
should refer to http://go/ndk-windows-vm for slightly more up-to-date Google-
specific setup instructions, but http://go/windows-cloudtop may be easier.

Windows testing can be done on Windows VMs in Google Compute Engine. To create
one:

* Install the [Google Cloud SDK](https://cloud.google.com/sdk/).
* Run `scripts/create_windows_instance.py $PROJECT_NAME $INSTANCE_NAME`
    * The project name is the name of the project you configured for the VMs.
    * The instance name is whatever name you want to use for the VM.

This process will create a `secrets.py` file in the NDK project directory that
contains the connection information.

The VM will have Chrome and Git installed and WinRM will be configured for
remote command line access.

TODO: Implement `run_tests.py --remote-build`.