src/compatibility/cts/development.jd

page.title=CTS Development
@jd:body

<!--
    Copyright 2015 The Android Open Source Project

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
-->
<div id="qv-wrapper">
  <div id="qv">
    <h2>In this document</h2>
    <ol id="auto-toc">
    </ol>
  </div>
</div>

<h2 id="initializing-your-repo-client">Initializing your Repo client</h2>
<p>Follow the <a href="{@docRoot}source/downloading.html">instructions</a>
to get and build the Android source code but specify a particular CTS branch
name, for example<code>-b android-5.0_r2</code>, when issuing the <code>repo
init</code> command. This assures your CTS changes will be included in the
next CTS release and beyond.</p>

<h2 id="building-and-running-cts">Building and running CTS</h2>

<p>Execute the following commands to build CTS and start the interactive
CTS console:</p>
<p class="note"><strong>Note:</strong> You may supply one of these other values
for <code>TARGET_PRODUCT</code> to build for different architectures:
<code>aosp_x86_64</code> or <code>aosp_mips</code></p>
<pre><code>cd <em>/path/to/android/root</em>
make cts -j32 TARGET_PRODUCT=aosp_arm64
cts-tradefed
</code></pre>

<p>At the cts-tf console, enter e.g.:</p>
<pre><code>run cts --plan CTS
</code></pre>

<h2 id="writing-cts-tests">Writing CTS tests</h2>

<p>CTS tests use JUnit and the Android testing APIs. Review the
<a href="https://developer.android.com/tools/testing/testing_android.html">Testing and Instrumentation</a>
tutorial while perusing the existing tests under the
<code>cts/tests</code> directory. You will see that CTS tests mostly follow the same
conventions used in other Android tests.</p>

<p>Since CTS runs across many production devices, the tests must follow
these rules:</p>
<ul>
<li>Must take into account varying screen sizes, orientations, and keyboard layouts.</li>
<li>Only use public API methods. In other words, avoid all classes, methods, and fields that are annotated with the "hide" annotation.</li>
<li>Avoid relying upon particular view layouts or depend on the dimensions of assets that may not be on some device.</li>
<li>Don't rely upon root privileges.</li>
</ul>

<h3 id="test-naming-and-location">Test naming and location</h3>

<p> Most CTS test cases target a specific class in the Android API. These tests
have Java package names with a <code>cts</code> suffix and class names with the
<code>Test</code> suffix. Each test case consists of multiple tests, where each
test usually exercises a particular method of the class being tested.
These tests are arranged in a directory structure where tests are grouped into
different categories such as "widgets" and "views." </p>

<p>
For example, the CTS test for the Java package
<code>android.widget.TextView</code> is
<code>android.widget.cts.TextViewTest</code> with its Java package name as
<code>android.widget.cts</code> and its class name as
<code>TextViewTest</code>.  </p>

<ul>
<li><strong>Java package name</strong><br/>The Java package name for the CTS tests
is the package name of the class that the test is testing, followed by ".cts".
For our example, the package name would be <code>android.widget.cts</code>.

<li><strong>Class name</strong><br/>The class name for CTS tests is <strong>
</strong>name of the class that the test is testing with "Test" appended. (For
example,  if a test is targeting <code>TextView</code>, the class name should be
<code>TextViewTest</code>.)

<li><strong>Module name (CTS v2 only)</strong><br/>CTS v2 organizes tests by module.
 The module name is usually the second string of the Java package name (in our
example, <code>widget</code>), although it does not have to be.</li>
</ul>

<p>
The directory structure and sample code depend on whether you are using CTS v1
or CTS v2.
</p>
<h4 id="cts-v1">CTS v1</h4>
<p>
For Android 6.0 and earlier, use CTS v1.  For CTS v1, the sample code is at
<code>cts/tests/tests/example</code>.
</p>
<p>
The directory structure in CTS v1 tests looks like this:
</p>
<pre class="no-pretty-print">
cts/
  tests/
    tests/
      <em>package-name</em>/
        Android.mk
        AndroidManifest.xml
        src/
          android/
            <em>package-name</em>/
              SampleDeviceActivity.java
              cts/
                SampleDeviceTest.java
</pre>
<h4 id="cts-v2"><strong>CTS v2</strong></h4>
<p>
For Android 7.0 and later, use CTS v2.  For details, see <a
href="https://android.googlesource.com/platform/cts/+/master/tests/sample/">the
sample test in AOSP</a>.
</p>
<p>
The CTS v2 directory structure looks like this:
</p>

<pre class="no-pretty-print">
cts/
  tests/
    <em>module-name</em>/
      Android.mk
      AndroidManifest.xml
      src/
        android/
          <em>package-name</em>/
            SampleDeviceActivity.java
            cts/
              SampleDeviceTest.java
</pre>

<h3 id="new-sample-packages">New sample packages</h3>

<p>When adding new tests, there may not be an existing directory to place your
test.  In those cases, you'll need to create the directory and copy the
appropriate sample files.</p>

<h4 id="cts-v1">CTS v1</h4>

<p> If you are using CTS v1, refer to the example under
<code>cts/tests/tests/example</code> and create a new directory. Also,
make sure to add your new package's module name from its <code>Android.mk</code>
to <code>CTS_COVERAGE_TEST_CASE_LIST</code>  in
<code>cts/CtsTestCaseList.mk</code>. This Makefile is used by
<code>build/core/tasks/cts.mk</code> to combine all the tests together to create
the final CTS package.  </p>

<h4 id="cts-v2">CTS v2</h4>
<p>
Use the sample test
<code><a href="https://android.googlesource.com/platform/cts/+/master/tests/sample/">/cts/tests/sample/</a></code>
to quick start your new test module with following steps:
</p>

<ol>
  <li>Run this command to create the test directory and copy sample files to it:
    <pre class="no-pretty-print">$ mkdir cts/tests/<i>module-name</i> && cp -r cts/tests/sample/* cts/tests/<i>module-name</i></pre>
  <li>Navigate to <code>cts/tests/<em>module-name</em></code> and substitute all instances of
"[Ss]ample" following the recommended naming convention from above.
  <li>Update <code>SampleDeviceActivity</code> to exercise the feature you're testing.
  <li>Update <code>SampleDeviceTest</code> to ensure the activity succeeds or logs its
errors.</li>
</ol>

<h4><strong>Additional directories</strong></h4>

<p> Other Android directories such as <code>assets</code>, <code>jni</code>,
<code>libs</code> and <code>res</code> can also be added. To add JNI code,
create a directory in the root of the project next to <code>src</code> with the native
code and an <code>Android.mk</code> in it.</p>

<p>
The makefile typically contains the following settings:
</p>
<pre>
LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := libCtsSample_jni

# don't include this package in any target
LOCAL_MODULE_TAGS := optional
LOCAL_SRC_FILES := <i>list of source code files</i>
LOCAL_C_INCLUDES := $(JNI_H_INCLUDE)

# Tag this module as a cts test artifact
LOCAL_COMPATIBILITY_SUITE := cts
LOCAL_SHARED_LIBRARIES := libnativehelper
LOCAL_SDK_VERSION := current
include $(BUILD_SHARED_LIBRARY)
</pre>

<h4>Android.mk file</h4>

<p> Finally, the <code>Android.mk</code> file in the root of the project will
need to be modified to build the native code and depend on it, as shown
below:</p>

<pre>
# All tests should include android.test.runner.
LOCAL_JAVA_LIBRARIES := android.test.runner

# Includes the jni code as a shared library
LOCAL_JNI_SHARED_LIBRARIES := libCtsSample_jni

# Include for InstrumentationCtsTestRunner
LOCAL_STATIC_JAVA_LIBRARIES := ctstestrunner...
LOCAL_SDK_VERSION := currentinclude $(BUILD_CTS_PACKAGE)

#Tells make to look in subdirectories for more make files to include
include $(call all-makefiles-under,$(LOCAL_PATH))
</pre>

<h2 id="Fix-remove-tests">Fix or remove tests</h2>
<p>Besides adding new tests there are other ways to contribute to CTS: Fix or
remove tests annotated with "BrokenTest" or "KnownFailure."</p>

<h2 id="submitting-your-changes">Submitting your changes</h2>
<p>Follow the <a href="{@docRoot}source/submit-patches.html">Submitting Patches workflow</a>
to contribute changes to CTS. A reviewer
will be assigned to your change, and your change should be reviewed shortly!</p>

<h2 id="release-schedule">Release schedule and branch information</h2>

<p>CTS releases follow this schedule.</p>

<p class="note"><strong>Note</strong>: This schedule is tentative and may be
updated from time to time as CTS for the given Android version matures.</p>

<table>
<tr>
  <th>Version</th>
  <th>Branch</th>
  <th>Frequency</th>
</tr>
</thead>
<tbody>
<tr>
  <td>7.0</td>
    <td>nougat-cts-dev</td>
      <td>Monthly</td>
      </tr>
<tr>
  <td>6.0</td>
  <td>marshmallow-cts-dev</td>
  <td>Monthly</td>
</tr>
<tr>
  <td>5.1</td>
  <td>lollipop-mr1-cts-dev</td>
  <td>Monthly</td>
</tr>
<tr>
  <td>5.0</td>
  <td>lollipop-cts-dev</td>
  <td>No releases planned</td>
</tr>
<tr>
  <td>4.4</td>
  <td>kitkat-cts-dev</td>
  <td>No releases planned</td>
</tr>
<tr>
  <td>4.3</td>
  <td>jb-mr2-cts-dev</td>
  <td>No releases planned</td>
</tr>
<tr>
  <td>4.2</td>
  <td>jb-mr1.1-cts-dev</td>
  <td>No releases planned</td>
</tr>
</table>

<h3 id="important-dates">Important Dates during month of the release</h3>

<ul>
  <li><strong>End of 1st Week</strong>: Code Freeze.  At this point,
submissions on the current branch will no longer be accepted and will not be
included in the next version of CTS.  Once we have chosen a candidate for
release, the branch will again be open and accepting new submissions.

  <li><strong>Second or third week</strong>: CTS is published in the Android
Open Source Project (AOSP).
</ul>

<h3 id="auto-merge">Auto-merge flow</h3>

<p>CTS development branches have been set up so that changes submitted to each
branch will automatically merge as below:<br>
jb-dev-> jb-mr1.1-cts-dev -> jb-mr2-cts-dev  -> kitkat-cts-dev ->
lollipop-cts-dev -> lollipop-mr1-cts-dev -> marshmallow-cts-dev ->
nougat-cts-dev -> &lt;private-development-branch for Android N MR1&gt;</p>

<p>If a changelist (CL) fails to merge correctly, the author of the CL will get
an email with instructions on how to resolve the conflict. In most of the
cases, the author of the CL can use the instructions to skip the auto-merge of
the conflicting CL.</p>