# Metalava

(Also known as "doclava2", but deliberately not named doclava2 since crucially
it does not generate docs; it's intended only for **meta**data extraction and
generation.)

Metalava is a metadata generator intended for the Android source tree, used for
a number of purposes:

* Allow extracting the API (into signature text files, into stub API files
  (which in turn get compiled into android.jar, the Android SDK library) and
  more importantly to hide code intended to be implementation only, driven by
  javadoc comments like @hide, @$doconly, @removed, etc, as well as various
  annotations.

* Extracting source level annotations into external annotations file (such as
  the typedef annotations, which cannot be stored in the SDK as .class level
  annotations).

* Diffing versions of the API and determining whether a newer version is
  compatible with the older version.

## Building and running

To build:

    $ ./gradlew

This builds a binary distribution in `../../out/host/common/install/metalava/bin/metalava`.

To run metalava:

    $ ../../out/host/common/install/metalava/bin/metalava
                    _        _
     _ __ ___   ___| |_ __ _| | __ ___   ____ _
    | '_ ` _ \ / _ \ __/ _` | |/ _` \ \ / / _` |
    | | | | | |  __/ || (_| | | (_| |\ V / (_| |
    |_| |_| |_|\___|\__\__,_|_|\__,_| \_/ \__,_|

    metalava extracts metadata from source code to generate artifacts such as the
    signature files, the SDK stub files, external annotations etc.

    Usage: metalava <flags>

    Flags:

    --help                                This message.
    --quiet                               Only include vital output
    --verbose                             Include extra diagnostic output

    ...

(*output truncated*)

Metalava has a new command line syntax, but it also understands the doclava1
flags and translates them on the fly. Flags that are ignored are listed on the
command line. If metalava is dropped into an Android framework build for
example, you'll see something like this (unless running with --quiet) :

    metalava: Ignoring javadoc-related doclava1 flag -J-Xmx1600m
    metalava: Ignoring javadoc-related doclava1 flag -J-XX:-OmitStackTraceInFastThrow
    metalava: Ignoring javadoc-related doclava1 flag -XDignore.symbol.file
    metalava: Ignoring javadoc-related doclava1 flag -doclet
    metalava: Ignoring javadoc-related doclava1 flag -docletpath
    metalava: Ignoring javadoc-related doclava1 flag -templatedir
    metalava: Ignoring javadoc-related doclava1 flag -htmldir
    ...

## Features

* Compatibility with doclava1: in compat mode, metalava spits out the same
  signature files for the framework as doclava1.

* Ability to read in an existing android.jar file instead of from source, which
  means we can regenerate signature files etc for older versions according to
  new formats (e.g. to fix past errors in doclava, such as annotation instance
  methods which were accidentally not included.)

* Ability to merge in data (annotations etc) from external sources, such as
  IntelliJ external annotations data as well as signature files containing
  annotations. This isn't just merged at export time, it's merged at codebase
  load time such that it can be part of the API analysis.

* Support for an updated signature file format (which is described in FORMAT.md)

  * Address errors in the doclava1 format which for example was missing
    annotation class instance methods

  * Improve the signature format such that it for example labels enums "enum"
    instead of "abstract class extends java.lang.Enum", annotations as
    "@interface" instead of "abstract class extends java.lang.Annotation", sorts
    modifiers in the canonical modifier order, using "extends" instead of
    "implements" for the superclass of an interface, and many other similar
    tweaks outlined in the `Compatibility` class. (Metalava also allows (and
    ignores) block comments in the signature files.)

  * Add support for writing (and reading) annotations into the signature
    files. This is vital now that some of these annotations become part of the
    API contract (in particular nullness contracts, as well as parameter names
    and default values.)

  * Support for a "compact" nullness format -- one based on Kotlin's
    syntax. Since the goal is to have **all** API elements explicitly state
    their nullness contract, the signature files would very quickly become
    bloated with @NonNull and @Nullable annotations everywhere. So instead, the
    signature format now uses a suffix of `?` for nullable, `!` for not yet
    annotated, and nothing for non-null.

    Instead of

        method public java.lang.Double convert0(java.lang.Float);
        method @Nullable public java.lang.Double convert1(@NonNull java.lang.Float);

    we have

        method public java.lang.Double! convert0(java.lang.Float!);
        method public java.lang.Double? convert1(java.lang.Float);

  * Other compactness improvements: Skip packages in some cases both for export
    and reinsert during import. Specifically, drop "java.lang."  from package
    names such that you have

        method public void onUpdate(int, String);

    instead of

        method public void onUpdate(int, java.lang.String);

    Similarly, annotations (the ones considered part of the API; unknown
    annotations are not included in signature files) use just the simple name
    instead of the full package name, e.g. `@UiThread` instead of
    `@android.annotation.UiThread`.

  * Misc documentation handling; for example, it attempts to fix sentences that
    javadoc will mistreat, such as sentences that "end" with "e.g. ".  It also
    looks for various common typos and fixes those; here's a sample error
    message running metalava on master: Enhancing docs:

        frameworks/base/core/java/android/content/res/AssetManager.java:166: error: Replaced Kitkat with KitKat in documentation for Method android.content.res.AssetManager.getLocales() [Typo]
        frameworks/base/core/java/android/print/PrinterCapabilitiesInfo.java:122: error: Replaced Kitkat with KitKat in documentation for Method android.print.PrinterCapabilitiesInfo.Builder.setColorModes(int, int) [Typo]

* Built-in support for injecting new annotations for use by the Kotlin compiler,
  not just nullness annotations found in the source code and annotations merged
  in from external sources, but also inferring whether nullness annotations have
  recently changed and if so marking them as @Migrate (which lets the Kotlin
  compiler treat errors in the user code as warnings instead of errors.)

* Support for generating documentation into the stubs files (so we can run
  javadoc or [Dokka](https://github.com/Kotlin/dokka) on the stubs files instead
  of the source code). This means that the documentation tool itself does not
  need to be able to figure out which parts of the source code is included in
  the API and which one is implementation; it is simply handed the filtered API
  stub sources that include documentation.

* Support for parsing Kotlin files. API files can now be implemented in Kotlin
  as well and metalava will parse and extract API information from them just as
  is done for Java files.

* Like doclava1, metalava can diff two APIs and warn about API compatibility
  problems such as removing API elements. Metalava adds new warnings around
  nullness, such as attempting to change a nullness contract incompatibly
  (e.g. you can change a parameter from non null to nullable for final classes,
  but not versa).  It also lets you diff directly on a source tree; it does not
  require you to create two signature files to diff.

* Consistent stubs: In doclava1, the code which iterated over the API and
  generated the signature files and generated the stubs had diverged, so there
  was some inconsistency. In metalava the stub files contain **exactly** the
  same signatures as in the signature files.

  (This turned out to be incredibly important; this revealed for example that
  StringBuilder.setLength(int) was missing from the API signatures since it is a
  public method inherited from a package protected super class, which the API
  extraction code in doclava1 missed, but accidentally included in the SDK
  anyway since it packages package private classes. Metalava strictly applies
  the exact same API as is listed in the signature files, and once this was
  hooked up to the build it immediately became apparent that it was missing
  important methods that should really be part of the API.)

* API Lint: Metalava can optionally (with --api-lint) run a series of additional
  checks on the public API in the codebase and flag issues that are discouraged
  or forbidden by the Android API Council; there are currently around 80 checks.
  Some of these take advantage of looking at the source code which wasn't
  possible with the signature-file based Python version; for example, it looks
  inside method bodies to see if you're synchronizing on this or the current
  class, which is forbidden.

* Baselines: Metalava can report all of its issues into a "baseline" file, which
  records the current set of issues. From that point forward, when metalava
  finds a problem, it will only be reported if it is not already in the
  baseline.  This lets you enforce new issues going forward without having to
  fix all existing violations. Periodically, as older issues are fixed, you can
  regenerate the baseline. For issues with some false positives, such as API
  Lint, being able to check in the set of accepted or verified false positives
  is quite important.

* Metalava can generate reports about nullness annotation coverage (which helps
  target efforts since we plan to annotate the entire API). First, it can
  generate a raw count:

        Nullness Annotation Coverage Statistics:
        1279 out of 46900 methods were annotated (2%)
        2 out of 21683 fields were annotated (0%)
        2770 out of 47492 parameters were annotated (5%)

  More importantly, you can also point it to some existing compiled applications
  (.class or .jar files) and it will then measure the annotation coverage of the
  APIs used by those applications. This lets us target the most important APIs
  that are currently used by a corpus of apps and target our annotation efforts
  in a targeted way. For example, running the analysis on the current version of
  framework, and pointing it to the
  [Plaid](https://github.com/nickbutcher/plaid) app's compiled output with

      ... --annotation-coverage-of ~/plaid/app/build/intermediates/classes/debug

  This produces the following output:

    324 methods and fields were missing nullness annotations out of 650 total
    API references.  API nullness coverage is 50%

    ```
    | Qualified Class Name                                         |      Usage Count |
    |--------------------------------------------------------------|-----------------:|
    | android.os.Parcel                                            |              146 |
    | android.view.View                                            |              119 |
    | android.view.ViewPropertyAnimator                            |              114 |
    | android.content.Intent                                       |              104 |
    | android.graphics.Rect                                        |               79 |
    | android.content.Context                                      |               61 |
    | android.widget.TextView                                      |               53 |
    | android.transition.TransitionValues                          |               49 |
    | android.animation.Animator                                   |               34 |
    | android.app.ActivityOptions                                  |               34 |
    | android.view.LayoutInflater                                  |               31 |
    | android.app.Activity                                         |               28 |
    | android.content.SharedPreferences                            |               26 |
    | android.content.SharedPreferences.Editor                     |               26 |
    | android.text.SpannableStringBuilder                          |               23 |
    | android.view.ViewGroup.MarginLayoutParams                    |               21 |
    | ... (99 more items                                           |                  |
    ```

Top referenced un-annotated members:

    ```
    | Member                                                       |      Usage Count |
    |--------------------------------------------------------------|-----------------:|
    | Parcel.readString()                                          |               62 |
    | Parcel.writeString(String)                                   |               62 |
    | TextView.setText(CharSequence)                               |               34 |
    | TransitionValues.values                                      |               28 |
    | View.getContext()                                            |               28 |
    | ViewPropertyAnimator.setDuration(long)                       |               26 |
    | ViewPropertyAnimator.setInterpolator(android.animation.Ti... |               26 |
    | LayoutInflater.inflate(int, android.view.ViewGroup, boole... |               23 |
    | Rect.left                                                    |               22 |
    | Rect.top                                                     |               22 |
    | Intent.Intent(android.content.Context, Class<?>)             |               21 |
    | Rect.bottom                                                  |               21 |
    | TransitionValues.view                                        |               21 |
    | VERSION.SDK_INT                                              |               18 |
    | Context.getResources()                                       |               18 |
    | EditText.getText()                                           |               18 |
    | ... (309 more items                                          |                  |
    ```

  From this it's clear that it would be useful to start annotating
  android.os.Parcel and android.view.View for example where there are
  unannotated APIs that are frequently used, at least by this app.

* Built on top of a full, type-resolved AST. Doclava1 was integrated with
  javadoc, which meant that most of the source tree was opaque. Therefore, as
  just one example, the code which generated documentation for typedef constants
  had to require the constants to all share a single prefix it could look
  for. However, in metalava, annotation references are available at the AST
  level, so it can resolve references and map them back to the original field
  references and include those directly.

* Support for extracting annotations. Metalava can also generate the external
  annotation files needed by Studio and lint in Gradle, which captures the
  typedefs (@IntDef and @StringDef classes) in the source code. Prior to this
  this was generated manually via the development/tools/extract code. This also
  merges in manually curated data; some of this is in the manual/ folder in this
  project.

* Support for extracting API levels (api-versions.xml). This was generated by
  separate code (tools/base/misc/api-generator), invoked during the build. This
  functionality is now rolled into metalava, which has one very important
  attribute: metalava will use this information when recording API levels for
  API usage. (Prior to this, this was based on signature file parsing in
  doclava, which sometimes generated incorrect results. Metalava uses the
  android.jar files themselves to ensure that it computes the exact available
  SDK data for each API level.)

* Misc other features. For example, if you use the @VisibleForTesting annotation
  from the support library, where you can express the intended visibility if the
  method had not required visibility for testing, then metalava will treat that
  method using the intended visibility instead when generating signature files
  and stubs.

## Architecture & Implementation

Metalava is implemented on top of IntelliJ parsing APIs (PSI and UAST). However,
these are hidden behind a "model": an abstraction layer which only exposes high
level concepts like packages, classes and inner classes, methods, fields, and
modifier lists (including annotations).

This is done for multiple reasons:

(1) It allows us to have multiple "back-ends": for example, metalava can read in
    a model not just from parsing source code, but from reading older SDK
    android.jar files (e.g. backed by bytecode) or reading previous signature
    files.  Reading in multiple versions of an API lets doclava perform
    "diffing", such as warning if an API is changing in an incompatible way. It
    can also generate signature files in the new format (including data that was
    missing in older signature files, such as annotation methods) without having
    to parse older source code which may no longer be easy to parse.

(2) There's a lot of logic for deciding whether code found in the source tree
    should be included in the API. With the model approach we can build up an
    API and for example mark a subset of its methods as included. By having a
    separate hierarchy we can easily perform this work once and pass around our
    filtered model instead of passing around PsiClass and PsiMethod instances
    and having to keep the filtered data separately and remembering to always
    consult the filter, not the PSI elements directly.

The basic API element class is "Item". (In doclava1 this was called a
"DocInfo".)  There are several sub interfaces of Item: PackageItem, ClassItem,
MemberItem, MethodItem, FieldItem, ParameterItem, etc. And then there are
several implementation hierarchies: One is PSI based, where you point metalava
to a source tree or a .jar file, and it constructs Items built on top of PSI:
PsiPackageItem, PsiClassItem, PsiMethodItem, etc. Another is textual, based on
signature files: TextPackageItem, TextClassItem, and so on.

The "Codebase" class captures a complete API snapshot (including classes that
are hidden, which is why it's called a "Codebase" rather than an "API").

There are methods to load codebases - from source folders, from a .jar file,
from a signature file. That's how API diffing is performed: you load two
codebases (from whatever source you want, typically a previous API signature
file and the current set of source folders), and then you "diff" the two.

There are several key helpers that help with the implementation, detailed next.

### Visiting Items

First, metalava provides an ItemVisitor. This lets you visit the API easily.
For example, here's how you can visit every class:

    coebase.accept(object : ItemVisitor() {
        override fun visitClass(cls: ClassItem) {
            // code operating on the class here
        }
    })

Similarly you can visit all items (regardless of type) by overriding
`visitItem`, or to specifically visit methods, fields and so on overriding
`visitPackage`, `visitClass`, `visitMethod`, etc.

There is also an `ApiVisitor`. This is a subclass of the `ItemVisitor`, but
which limits itself to visiting code elements that are part of the API.

This is how for example the SignatureWriter and the StubWriter are both
implemented: they simply extend `ApiVisitor`, which means they'll only export
the API items in the codebase, and then in each relevant method they emit the
signature or stub data:

    class SignatureWriter(
            private val writer: PrintWriter,
            private val generateDefaultConstructors: Boolean,
            private val filter: (Item) -> Boolean) : ApiVisitor(
            visitConstructorsAsMethods = false) {

    ....

    override fun visitConstructor(constructor: ConstructorItem) {
        writer.print("    ctor ")
        writeModifiers(constructor)
        writer.print(constructor.containingClass().fullName())
        writeParameterList(constructor)
        writeThrowsList(constructor)
        writer.print(";\n")
    }

    ....

### Visiting Types

There is a `TypeVisitor` similar to `ItemVisitor` which you can use to visit all
types in the codebase.

When computing the API, all types that are included in the API should be
included (e.g. if `List<Foo>` is part of the API then `Foo` must be too).  This
is easy to do with the `TypeVisitor`.

### Diffing Codebases

Another visitor which helps with implementation is the ComparisonVisitor:

    open class ComparisonVisitor {
        open fun compare(old: Item, new: Item) {}
        open fun added(item: Item) {}
        open fun removed(item: Item) {}

        open fun compare(old: PackageItem, new: PackageItem) { }
        open fun compare(old: ClassItem, new: ClassItem) { }
        open fun compare(old: MethodItem, new: MethodItem) { }
        open fun compare(old: FieldItem, new: FieldItem) { }
        open fun compare(old: ParameterItem, new: ParameterItem) { }

        open fun added(item: PackageItem) { }
        open fun added(item: ClassItem) { }
        open fun added(item: MethodItem) { }
        open fun added(item: FieldItem) { }
        open fun added(item: ParameterItem) { }

        open fun removed(item: PackageItem) { }
        open fun removed(item: ClassItem) { }
        open fun removed(item: MethodItem) { }
        open fun removed(item: FieldItem) { }
        open fun removed(item: ParameterItem) { }
    }

This makes it easy to perform API comparison operations.

For example, metalava has a feature to mark "newly annotated" nullness
annotations as migrated. To do this, it just extends `ComparisonVisitor`,
overrides the `compare(old: Item, new: Item)` method, and checks whether the old
item has no nullness annotations and the new one does, and if so, also marks the
new annotations as @Migrate.

Similarly, the API Check can simply override

    open fun removed(item: Item) {
        reporter.report(error, item, "Removing ${Item.describe(item)} is not allowed")
    }

to flag all API elements that have been removed as invalid (since you cannot
remove API. (The real check is slightly more complicated; it looks into the
hierarchy to see if there still is an inherited method with the same signature,
in which case the deletion is allowed.))

### Documentation Generation

As mentioned above, metalava generates documentation directly into the stubs
files, which can then be processed by Dokka and Javadoc to generate the same
docs as before.

Doclava1 was integrated with javadoc directly, so the way it generated metadata
docs (such as documenting permissions, ranges and typedefs from annotations) was
to insert auxiliary tags (`@range`, `@permission`, etc) and then this would get
converted into English docs later via `macros_override.cs`.

This it not how metalava does it; it generates the English documentation
directly. This was not just convenient for the implementation (since metalava
does not use javadoc data structures to pass maps like the arguments for the
typedef macro), but should also help Dokka -- and arguably the Kotlin code which
generates the documentation is easier to reason about and to update when it's
handling loop conditionals. (As a result I for example improved some of the
grammar, e.g. when it's listing a number of possible constants the conjunction
is usually "or", but if it's a flag, the sentence begins with "a combination of
" and then the conjunction at the end should be "and").