src/compatibility/cts/index.jd

page.title=Compatibility Test Suite
@jd:body

<!--
    Copyright 2015 The Android Open Source Project

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
-->
<div id="qv-wrapper">
  <div id="qv">
    <h2>In this document</h2>
    <ol id="auto-toc">
    </ol>
  </div>
</div>

<h2 id="how-does-the-cts-work">How does the CTS work?</h2>

<p>The Compatibility Test Suite (CTS) is a free, commercial-grade test suite,
available for <a href="downloads.html">download</a>.  The CTS represents the
"mechanism" of compatibility.</p>

<p>The CTS runs on a desktop machine and executes test cases directly on
attached devices or an emulator. The CTS is a set of unit tests designed to be
integrated into the daily workflow (such as via a continuous build system) of
the engineers building a device. Its intent is to reveal incompatibilities
early on, and ensure that the software remains compatible throughout the
development process.</p>


<p>The CTS is an automated testing harness that includes two major software components:</p>
<ul>
<li>
<p>The CTS tradefed test harness runs on your desktop machine and manages test execution.</p>
</li>
<li>
<p>Individual test cases are executed on the Device Under Test (DUT). The test
cases are written in Java as JUnit tests and packaged as
Android .apk files to run on the actual device target.</p>
</li>
</ul>

<p>The Compatibility Test Suite Verifier (CTS Verifier) is a supplement to the
CTS available for <a href="downloads.html">download</a>. CTS Verifier
provides tests for APIs and functions that cannot be tested on a stationary
device without manual input (e.g. audio quality, accelerometer, etc).</p>

<p>The CTS Verifier is a tool for manual testing and includes the following software components:</p>
<ul>
<li>
<p>The CTS verifier app that is executed on the DUT and collects the results.<p>
</li>
<li>
<p>The executable(s) or script(s) that are executed on the desktop machine to
provide data or additional control for some test cases in the CTS Verifier
app.</p>
</li>
</ul>

<h2 id="workflow">Workflow</h2>

<div class="figure" style="width:383px">
  <img src="images/cts-0.png" alt="CTS flow" height="340px" id="figure1" />
  <p class="img-caption">
    <strong>Figure 1.</strong> How to use CTS
  </p>
</div>

<p>This diagram summarizes CTS workflow. Please refer to the subpages of this
section starting with <a href="setup.html">Setup</a> for detailed
instructions.</p>

<h2 id="types-of-test-cases">Types of test cases</h2>
<p>The CTS includes the following types of test cases:</p>
<ul>
<li>
<p><em>Unit tests</em> test atomic units of code within the Android platform; e.g. a single class, such as java.util.HashMap.</p>
</li>
<li>
<p><em>Functional tests</em> test a combination of APIs together in a higher-level use-case.</p>
</li>
</ul>
<p>Future versions of the CTS will include the following types of test cases:</p>
<ul>
<li>
<p><em>Robustness tests</em> test the durability of the system under stress.</p>
</li>
<li>
<p><em>Performance tests</em> test the performance of the system against defined benchmarks, for example rendering frames per second.</p>
</li>
</ul>
<h2 id="areas-covered">Areas covered</h2>
<p>The unit test cases cover the following areas to ensure compatibility:</p>
<table>
<thead>
<tr>
<th>Area</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>Signature tests</td>
<td>For each Android release, there are XML files describing the signatures of all public APIs contained in the release. The CTS contains a utility to check those API signatures against the APIs available on the device. The results from signature checking are recorded in the test result XML file.</td>
</tr>
<tr>
<td>Platform API Tests</td>
<td>Test the platform (core libraries and Android Application Framework) APIs as documented in the SDK <a href="https://developer.android.com/reference/classes.html">Class Index</a> to ensure API correctness, including correct class, attribute and method signatures, correct method behavior, and negative tests to ensure expected behavior for incorrect parameter handling.</td>
</tr>
<tr>
<td>Dalvik Tests</td>
<td>The tests focus on testing the Dalvik Executable Format.</td>
</tr>
<tr>
<td>Platform Data Model</td>
<td>The CTS tests the core platform data model as exposed to application developers through content providers, as documented in the SDK <a href="https://developer.android.com/reference/android/provider/package-summary.html">android.provider</a> package: contacts, browser, settings, etc.</td>
</tr>
<tr>
<td>Platform Intents</td>
<td>The CTS tests the core platform intents, as documented in the SDK <a href="https://developer.android.com/guide/appendix/g-app-intents.html">Available Intents</a>.</td>
</tr>
<tr>
<td>Platform Permissions</td>
<td>The CTS tests the core platform permissions, as documented in the SDK <a href="https://developer.android.com/reference/android/Manifest.permission.html">Available Permissions</a>.</td>
</tr>
<tr>
<td>Platform Resources</td>
<td>The CTS tests for correct handling of the core platform resource types, as documented in the SDK <a href="https://developer.android.com/guide/topics/resources/available-resources.html">Available Resource Types</a>. This includes tests for: simple values, drawables, nine-patch, animations, layouts, styles and themes, and loading alternate resources.</td>
</tr>
</tbody>
</table>