docs/source/quality.rst

.. _tools:

********************************************
Testing and Ensuring Type Annotation Quality
********************************************

Testing Annotation Accuracy
===========================

When creating a package with type annotations, authors may want to validate
that the annotations they publish meet their expectations.
This is especially important for library authors, for whom the published
annotations are part of the public interface to their package.

There are several approaches to this problem, and this document will show
a few of them.

.. note::

    For simplicity, we will assume that type-checking is done with ``mypy``.
    Many of these strategies can be applied to other type-checkers as well.

Testing Using ``mypy --warn-unused-ignores``
--------------------------------------------

Clever use of ``--warn-unused-ignores`` can be used to check that certain
expressions are or are not well-typed.

The idea is to write normal python files which contain valid expressions along
with invalid expressions annotated with ``type: ignore`` comments. When
``mypy --warn-unused-ignores`` is run on these files, it should pass.
A directory of test files, ``typing_tests/``, can be maintained.

This strategy does not offer strong guarantees about the types under test, but
it requires no additional tooling.

If the following file is under test

.. code-block:: python

    # foo.py
    def bar(x: int) -> str:
        return str(x)

Then the following file tests ``foo.py``:

.. code-block:: python

    bar(42)
    bar("42")  # type: ignore [arg-type]
    bar(y=42)  # type: ignore [call-arg]
    r1: str = bar(42)
    r2: int = bar(42)  # type: ignore [assignment]

Checking ``reveal_type`` output from ``mypy.api.run``
-----------------------------------------------------

``mypy`` provides a subpackage named ``api`` for invoking ``mypy`` from a
python process. In combination with ``reveal_type``, this can be used to write
a function which gets the ``reveal_type`` output from an expression. Once
that's obtained, tests can assert strings and regular expression matches
against it.

This approach requires writing a set of helpers to provide a good testing
experience, and it runs mypy once per test case (which can be slow).
However, it builds only on ``mypy`` and the test framework of your choice.

The following example could be integrated into a testsuite written in
any framework:

.. code-block:: python

    import re
    from mypy import api

    def get_reveal_type_output(filename):
        result = api.run([filename])
        stdout = result[0]
        match = re.search(r'note: Revealed type is "([^"]+)"', stdout)
        assert match is not None
        return match.group(1)


For example, we can use the above to provide a ``run_reveal_type`` pytest
fixture which generates a temporary file and uses it as the input to
``get_reveal_type_output``:

.. code-block:: python

    import os
    import pytest

    @pytest.fixture
    def _in_tmp_path(tmp_path):
        cur = os.getcwd()
        try:
            os.chdir(tmp_path)
            yield
        finally:
            os.chdir(cur)

    @pytest.fixture
    def run_reveal_type(tmp_path, _in_tmp_path):
        content_path = tmp_path / "reveal_type_test.py"

        def func(code_snippet, *, preamble = ""):
            content_path.write_text(preamble + f"reveal_type({code_snippet})")
            return get_reveal_type_output("reveal_type_test.py")

        return func


For more details, see `the documentation on mypy.api
<https://mypy.readthedocs.io/en/stable/extending_mypy.html#integrating-mypy-into-another-python-application>`_.

pytest-mypy-plugins
-------------------

`pytest-mypy-plugins <https://github.com/typeddjango/pytest-mypy-plugins>`_ is
a plugin for ``pytest`` which defines typing test cases as YAML data.
The test cases are run through ``mypy`` and the output of ``reveal_type`` can
be asserted.

This project supports complex typing arrangements like ``pytest`` parametrized
tests and per-test ``mypy`` configuration. It requires that you are using
``pytest`` to run your tests, and runs ``mypy`` in a subprocess per test case.

This is an example of a parametrized test with ``pytest-mypy-plugins``:

.. code-block:: yaml

    - case: with_params
      parametrized:
        - val: 1
          rt: builtins.int
        - val: 1.0
          rt: builtins.float
      main: |
        reveal_type({[ val }})  # N: Revealed type is '{{ rt }}'

Improving Type Completeness
===========================

One of the goals of many libraries is to ensure that they are "fully type
annotated", meaning that they provide complete and accurate type annotations
for all functions, classes, and objects. Having full annotations is referred to
as "type completeness" or "type coverage".

Here are some tips for increasing the type completeness score for your
library:

-  Make type completeness an output of your testing process. Several type
   checkers have options for generating useful output, warnings, or even
   reports.
-  If your package includes tests or sample code, consider removing them
   from the distribution. If there is good reason to include them,
   consider placing them in a directory that begins with an underscore
   so they are not considered part of your library’s interface.
-  If your package includes submodules that are meant to be
   implementation details, rename those files to begin with an
   underscore.
-  If a symbol is not intended to be part of the library’s interface and
   is considered an implementation detail, rename it such that it begins
   with an underscore. It will then be considered private and excluded
   from the type completeness check.
-  If your package exposes types from other libraries, work with the
   maintainers of these other libraries to achieve type completeness.

.. warning::

    The ways in which different type checkers evaluate and help you achieve
    better type coverage may differ. Some of the above recommendations may or
    may not be helpful to you, depending on which type checking tools you use.

``mypy`` disallow options
-------------------------

``mypy`` offers several options which can detect untyped code.
More details can be found in `the mypy documentation on these options
<https://mypy.readthedocs.io/en/latest/command_line.html#untyped-definitions-and-calls>`_.

Some basic usages which make ``mypy`` error on untyped data are::

    mypy --disallow-untyped-defs
    mypy --disallow-incomplete-defs

``pyright`` type verification
-----------------------------

pyright has a special command line flag, ``--verifytypes``, for verifying
type completeness. You can learn more about it from
`the pyright documentation on verifying type completeness
<https://github.com/microsoft/pyright/blob/main/docs/typed-libraries.md#verifying-type-completeness>`_.

``mypy`` reports
----------------

``mypy`` offers several options options for generating reports on its analysis.
See `the mypy documentation on report generation
<https://mypy.readthedocs.io/en/stable/command_line.html#report-generation>`_ for details.