docs/testability.md

# Testability

[TOC]

## How to write testable libraries

When developers use a Jetpack library, it should be easy to write reliable and
automated tests for their own code's functionality. In most cases, tests written
against a library will need to interact with that library in some way -- setting
up preconditions, reaching synchronization points, making calls -- and the
library should provide necessary functionality to enable such tests, either
through public APIs or optional `-testing` artifacts.

**Testability**, in this document, means how easily and effectively the users of
a library can create tests for apps that use that library.

NOTE Tests that check the behavior of a library have a different mission than
tests made by app developers using that library; however, library developers may
find themselves in a similar situation when writing tests for sample code.

Often, the specifics of testability will vary from library to library and there
is no one way of providing it. Some libraries have enough public API surface,
others provide additional testing artifacts (e.g.
[Lifecycle Runtime Testing artifact](https://maven.google.com/web/index.html?q=lifecycle#androidx.lifecycle:lifecycle-runtime-testing)).

The best way to check if your library is testable is to try to write a sample
app with tests. Unlike regular library tests, these apps will be limited to the
public API surface of the library.

Keep in mind that a good test for a sample app should have:

*   [No side effects](#side-effects)
*   [No dependencies on time / looper (except for UI)](#external-dependencies)
*   [No private API calls](#private-apis)
*   [No assumptions on undefined library behavior](#undefined-behavior)

If you are able to write such tests for your library, you are good to go. If you
struggled or found yourself writing duplicate testing code, there is room for
improvement.

To get started with sample code, see
[Sample code in Kotlin modules](api_guidelines.md#sample-code-in-kotlin-modules)
for information on writing samples that can be referenced from API reference
documentation or [Project directory structure](api_guidelines.md#module-structure)
for module naming guidelines if you'd like to create a basic test app.

### Avoiding side-effects {#side-effects}

#### Ensure proper scoping for your library a.k.a. Avoid Singletons

Singletons are usually bad for tests as they live across different tests,
opening the gates for possible side-effects between tests. When possible, try to
avoid using singletons. If it is not possible, consider providing a test
artifact that will reset the state of the singleton between tests.

#### Side effects due to external resources

Sometimes, your library might be controlling resources on the device in which
case even if it is not a singleton, it might have side-effects. For instance,
Room, being a database library, inherently modifies a file on the disk. To allow
proper isolated testing, Room provides a builder option to create the database
[in memory](https://developer.android.com/reference/androidx/room/Room#inMemoryDatabaseBuilder\(android.content.Context,%20java.lang.Class%3CT%3E\))
A possible alternative solution could be to provide a test rule that will
properly close databases after each test.

If your library needs an inherently singleton resource (e.g. `WorkManager` is a
wrapper around `JobScheduler` and there is only 1 instance of it provided by the
system), consider providing a testing artifact. To provide isolation for tests,
the WorkManager library ships a
[separate testing artifact](https://developer.android.com/topic/libraries/architecture/workmanager/how-to/integration-testing)

### "External" dependencies {#external-dependencies}

#### Allow configuration of external resource dependencies

A common example of this use case is libraries that do multi-threaded
operations. For Kotlin libraries, this is usually achieved by receiving a
coroutine context or scope. For Java libraries, it is commonly an `Executor`. If
you have a case like this, make sure it can be passed as a parameter to your
library.

NOTE Android API Guidelines require that methods accepting a callback
[must also take an Executor](https://android.googlesource.com/platform/developers/docs/+/refs/heads/master/api-guidelines/index.md#provide-executor)

For example, the Room library allows developers to
[pass different executors](https://developer.android.com/reference/androidx/room/RoomDatabase.Builder#setQueryExecutor\(java.util.concurrent.Executor\))
for background query operations. When writing a test, developers can invoke this
with a custom executor where they can track work completion.

*   [sample test](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:room/integration-tests/kotlintestapp/src/androidTest/java/androidx/room/integration/kotlintestapp/test/SuspendingQueryTest.kt;l=672)

If the external resource you require does not make sense as a public API, such
as a main thread executor, then you can provide a testing artifact which will
allow setting it. For example, the Lifecycle package depends on the main thread
executor to function but for an application, customizing it does not make sense
(as there is only 1 "pre-defined" main thread for an app). For testing purposes,
the Lifecycle library provides a testing artifact which includes
[TestRules](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:arch/core-testing/src/main/java/androidx/arch/core/executor/testing/CountingTaskExecutorRule.java)
to change them.

#### Fakes for external dependencies

Sometimes, the developer might want to track side effects of your library for
end to end testing. For instance, if your library provides some functionality
that might decide to toggle bluetooth, outside developer's direct control, it
might be a good idea to have an interface for that functionality and also
provide a fake that can record such calls. If you don't think that interface
makes sense as a library configuration, you can
[restrict](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:annotation/annotation/src/main/java/androidx/annotation/RestrictTo.java)
that interface to your library group and provide a testing artifacts with the
fake so that developer can observe side effects only in tests while you can
avoid creating unnecessary APIs.

NOTE There is a fine balance between making a library configurable and making
configuration a nightmare for the developer. You should try to always have
defaults for these configurable objects to ensure your library is easy to use
while still testable when necessary.

### Avoiding the need for private API calls in tests {#private-apis}

#### Provide additional functionality for tests

There are certain situations where it could be useful to provide more APIs that
only make sense in the scope of testing. For example, a `Lifecycle` class has
methods that are bound to the main thread but developers may want to have other
tests that rely on lifecycle but do not run on the main thread (or even on an
emulator). For such cases, you may create APIs that are testing only to allow
developers to use them only in tests while giving them the confidence that it
will behave as close as possible to a real implementation. For the case above,
`LifecycleRegistry` provides an API to
[create](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:lifecycle/lifecycle-runtime/src/main/java/androidx/lifecycle/LifecycleRegistry.java;l=340)
an instance of it that will not enforce thread restrictions.

NOTE Even though the implementation referenced above is acceptable, it is always
better to create such functionality in additional testing artifacts when
possible.

### Avoiding assumptions in app code for library behavior {#undefined-behavior}

#### Provide fakes for common classes in a `-testing` artifact

In some cases, the developer might need an instance of a class provided by your
library but does not want to (or cannot) initiate it in tests. Moreover,
behavior of such classes might not be fully defined for edge cases, making it
difficult for developers to mock them.

For such cases, it is a good practice to provide a fake implementation out of
the box that can be controlled in tests. For example, the Lifecycle library
provides a
[fake implementation](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:lifecycle/lifecycle-runtime-testing/src/main/java/androidx/lifecycle/testing/TestLifecycleOwner.kt)
for the `LifecycleOwner` class that can be manipulated in tests to create
different use cases.

## Document how to test with your library

Even when your library is fully testable, it is often not clear for a developer
to know which APIs are safe to call in tests and when. Providing guidance on
[d.android.com](https://d.android.com) will make it much easier for the
developer to start testing with your library.

## Testability anti-patterns

When writing tests against your APIs, look out for the following anti-patterns.

### Calling `Instrumentation.waitForIdleSync()` as a synchronization barrier

The `waitForIdle()` and `waitForIdleSync()` methods claim to "(Synchronously)
wait for the application to be idle." and may seem like reasonable options when
there is no obvious way to observe whether an action has completed; however,
these methods know nothing about the context of the test and return when the
main thread's message queue is empty.

```java {.bad}
view.requestKeyboardFocus();
Instrumentation.waitForIdleSync();
sendKeyEvents(view, "1234");
// There is no guarantee that `view` has keyboard focus yet.
device.pressEnter();
```

In apps with an active UI animation, the message queue is _never empty_. If the
app is waiting for a callback across IPC, the message queue may be empty despite
the test not reaching the desired state.

In some cases, `waitForIdleSync()` introduces enough of a delay that unrelated
asynchronous actions happen to have completed by the time the method returns;
however, this delay is purely coincidental and eventually leads to flakiness.

Instead, find a reliable synchronization barrier that guarantees the expected
state has been reached or the requested action has been completed. This might
mean adding listeners, callbacks, `ListenableFuture`s, or `LiveData`.

See [Asynchronous work](api_guidelines.md#async) in the API Guidelines for more
information on exposing the state of asynchronous work to clients.