en/compatibility/vts/automated-test-infra.html - platform/docs/source.android.com - Git at Google

 <html devsite>
   <head>
     <title>Automated Testing Infrastructure</title>
     <meta name="project_path" value="/_project.yaml" />
     <meta name="book_path" value="/_book.yaml" />
   </head>
   <body>
   {% include "_versions.html" %}
   <!--
       Copyright 2018 The Android Open Source Project

       Licensed under the Apache License, Version 2.0 (the "License");
       you may not use this file except in compliance with the License.
       You may obtain a copy of the License at

           http://www.apache.org/licenses/LICENSE-2.0

       Unless required by applicable law or agreed to in writing, software
       distributed under the License is distributed on an "AS IS" BASIS,
       WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
       See the License for the specific language governing permissions and
       limitations under the License.
   -->

 <p>
   Android {{ androidPVersionNumber }} includes a Vendor Test Suite (VTS)
   infrastructure for automated testing of VTS, CTS, or other tests on partner
   devices running the AOSP generic system image (GSI). Previously, running these
   tests was a highly manual operation; the new VTS test infrastructure is
   designed to support automated testing multiple times a day on multiple
   devices.
 </p>

 <h2 id=architecture>Architecture</h2>

 <p>
   The VTS automated testing infrastructure uses the following architecture:
   </p>

 <p>
   <img src="images/vts-automated.png"
   alt="Automated test architecture"
   title="Automated test architecture">
 </p>
 <figcaption>
   <strong>Figure 1.</strong> VTS automated testing infrastructure architecture
 </figcaption>

 <p>
   When a test is triggered, the VTS automated testing infrastructure performs
   the following tasks:
 </p>

 <ol>
   <li>Fetches build artifacts and test resources from different locations:
   <ul>
     <li><strong>Partner Android Build (PAB)</strong>. For the GSI, VTS
     framework, and some other builds.</li>
     <li><strong>Local filesystem, Google Cloud Storage, or other vendor-specific
     build system</strong>. For partners who do not store builds in Google's
     cloud.</li>
   </ul>
   </li>
   <li>Flashes build artifacts (from the device) and the GSI (from AOSP) onto the
   connected device(s).</li>
   <li>Runs VTS tests using local TradeFed or a TradeFed in the cloud.</li>
   <li>Reports test results to the VTS dashboard</li>
 </ol>

 <p>
   The process is coordinated by the VTS host controller (HC), a machine in the
   lab that directs the behavior of all connected devices under test. The HC is
   responsible for fetching the latest builds, flashing them onto devices, and
   invoking tests (either locally or through the commander). It also communicates
   with a cloud scheduler and directs traffic between the scheduler and the
   TradeFed instance (or some other harness) running on the HC. For details on
   the host controller, see <a href="/compatibility/vts/host-controller">Host
   Controller Architecture</a>.
 </p>

 <h2 id="resource-providers">Resource providers</h2>

 <p>
   Automated testing requires resources such as system builds, test files, and
   VTS artifacts. While it's possible to build these from source, it is easier to
   build them from tip-of-tree regularly then post the artifacts for download.
 </p>

 <p>
   Partners can access automation resources using the following locations:
 </p>

 <ul>
   <li><strong>Partner Android Build</strong>. Programmatic access granted on a
   per-account basis.</li>
   <li><strong>Local filesystem</strong> (or similar). For partners who do not
   use the Partner Android Build.</li>
 </ul>

 <p>
   For use in flashing the devices later, resources include build providers for
   both options, extending from a single <code>build_provider.py</code> that
   stores the builds in local temporary directories.
 </p>

 <h3 id=partner-android-build>Partner Android Build</h3>

 <p>
   In Android 8.1 and lower releases, Android partners were required to visit the
   Partner Android Build website
   (<a href="https://partner.android.com/build" class="external">https://partner.android.com/build</a>),
   navigate to their account, and fetch the latest system images through the user
   interface. To help partners avoid this slow and labor-intensive process,
   Android {{ androidPVersionNumber }} includes support for automatically
   downloading these resources from PAB when
   the appropriate credentials are provided.
 </p>

 <h4 id="establishing-access">Establishing access</h4>

 <p>
   Programmatic access uses OAuth2 on Google APIs to access the required RPCs.
   Using the
   <a href="https://developers.google.com/api-client-library/python/guide/aaa_oauth#flow_from_clientsecrets" class="external">standard
   approach</a> for generating OAuth2 credentials, the partner must set up a
   client id/secret pair with Google. When the
   <code>PartnerAndroidBuildClient</code> is pointed to that secret for the first
   time, it opens a browser window for the user to log in to their Google
   account, which generates the OAuth2 credentials needed to move forward. The
   credentials (access token and refresh token) are stored locally, meaning
   partners should need to login only once.
 </p>

 <h4>POST request for URL</h4>

 <p>
   Clicking a resource link in PAB sends a POST request that includes the
   necessary data for that resource, including:
 </p>

 <ul>
   <li>build id, build target</li>
   <li>resource name</li>
   <li>branch</li>
   <li>release candidate name and whether or not the candidate is an internal
   build</li>
 </ul>

 <p>
   The POST request is received by the <code>downloadBuildArtifact</code> method
   of the <code>buildsvc</code> RPC, which returns a URL that can be used to
   access the resource.

 <ul>
   <li>For Clockwork Companion APK resources, the URL is a readable URL hosted on
   PAB (which is auth-protected and accessible with the appropriate OAuth2
   credentials).</li>
   <li>For other resources, the URL is long, non-protected URL from the internal
   Android Build API (which expires after five minutes).</li>
 </ul>

 <h4 id="getting-url">Getting the URL </h4>

 <p>
   To avoid cross-site request forgery, the <code>buildsvc</code> RPC requires an
   XSRF token to be POSTed with the other parameters. While this token makes the
   process more secure, it also makes programmatic access much harder since the
   token (which is available only in the JavaScript of the PAB page) is now also
   required for access.
 </p>

 <p>
   To avoid this issue, Android {{ androidPVersionNumber }} redesigns the URL
   naming scheme for all files (not just APKs) to use predictable URL names for
   accessing artifact lists and artifact URLs. The PAB now uses a convenient URL
   format that enables partners to download resources; HC scripts can download
   those APKs easily, since the URL format is known, and HC can bypass the
   XSRF/cookie issues because it does not need the <code>buildsvc</code> RPC.
 </p>

 <h3 id="local-filesystem">Local filesystem</h3>

 <p>
   Given a directory with a list (or zip file) of artifacts, the build provider
   sets the relevant images based on what's in the directory. You can use the
   <a href="https://cloud.google.com/storage/docs/gsutil" class="external">gsutil</a>
   tool to copy files from Google Cloud Storage to a local directory.
 </p>

 <h2 id="flashing-builds">Flashing builds</h2>

 <p>
   After the most recent device images are downloaded to the host, those images
   must be flashed onto the devices. This is done using the standard
   <code>adb</code> and <code>fastboot</code> commands and Python subprocesses,
   based on the temporary file paths stored by the build providers.
 </p>

 <p>
   Supported actions:
 </p>

 <ul>
   <li>Flashing only the GSI</li>
   <li>Flashing individual images from the main system (e.g.,
   <code>fastboot flash boot boot.img</code>)</li>
   <li>Flashing all images from the main system. Example:
   <ul>
     <li><code>fastboot flashall</code> (using the built-in <code>flashall</code>
     utility)</li>
     <li><code>fastboot flash</code> (one at a time)</li>
   </ul>
   </li>
 </ul>

 <h2 id="running=tests">Running tests</h2>

 <p>
   In Android {{ androidPVersionNumber }}, the VTS automated testing
   infrastructure supports only the TradeFed test harness but could be extended
   to support other harnesses in the future.
 </p>

 <p>
   After the devices are prepared, you can invoke tests using one of the
   following options:
 </p>

 <ul>
   <li>When using TradeFed locally, use the <code>test</code> command in the host
   controller, which takes the name of a VTS test plan (e.g.
   <code>vts-selftest</code>) and runs the test.</li>
   <li>When using a TradeFed Cluster (optionally connected to MTT), use the
   <code>lease</code> command in the host controller console, which looks for
   unfulfilled test runs.</li>
 </ul>

 <p>
   If using TradeFedCluster, TradeFed runs
   <a href="/compatibility/vts/host-controller">locally as a remote manager</a>.
   If not, the tests are invoked using Python subprocesses.
 </p>

 <h2 id="reporting-results">Reporting results</h2>

 <p>
   Test results are automatically reported to some VTS dashboard projects by
   <code>VtsMultiDeviceTest</code>.
 </p>

 </body>
 </html>
	<html devsite>
	<head>
	<title>Automated Testing Infrastructure</title>
	<meta name="project_path" value="/_project.yaml" />
	<meta name="book_path" value="/_book.yaml" />
	</head>
	<body>
	{% include "_versions.html" %}
	<!--
	Copyright 2018 The Android Open Source Project

	Licensed under the Apache License, Version 2.0 (the "License");
	you may not use this file except in compliance with the License.
	You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing, software
	distributed under the License is distributed on an "AS IS" BASIS,
	WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	See the License for the specific language governing permissions and
	limitations under the License.
	-->

	<p>
	Android {{ androidPVersionNumber }} includes a Vendor Test Suite (VTS)
	infrastructure for automated testing of VTS, CTS, or other tests on partner
	devices running the AOSP generic system image (GSI). Previously, running these
	tests was a highly manual operation; the new VTS test infrastructure is
	designed to support automated testing multiple times a day on multiple
	devices.
	</p>

	<h2 id=architecture>Architecture</h2>

	<p>
	The VTS automated testing infrastructure uses the following architecture:
	</p>

	<p>
	<img src="images/vts-automated.png"
	alt="Automated test architecture"
	title="Automated test architecture">
	</p>
	<figcaption>
	<strong>Figure 1.</strong> VTS automated testing infrastructure architecture
	</figcaption>

	<p>
	When a test is triggered, the VTS automated testing infrastructure performs
	the following tasks:
	</p>

	<ol>
	<li>Fetches build artifacts and test resources from different locations:
	<ul>
	<li><strong>Partner Android Build (PAB)</strong>. For the GSI, VTS
	framework, and some other builds.</li>
	<li><strong>Local filesystem, Google Cloud Storage, or other vendor-specific
	build system</strong>. For partners who do not store builds in Google's
	cloud.</li>
	</ul>
	</li>
	<li>Flashes build artifacts (from the device) and the GSI (from AOSP) onto the
	connected device(s).</li>
	<li>Runs VTS tests using local TradeFed or a TradeFed in the cloud.</li>
	<li>Reports test results to the VTS dashboard</li>
	</ol>

	<p>
	The process is coordinated by the VTS host controller (HC), a machine in the
	lab that directs the behavior of all connected devices under test. The HC is
	responsible for fetching the latest builds, flashing them onto devices, and
	invoking tests (either locally or through the commander). It also communicates
	with a cloud scheduler and directs traffic between the scheduler and the
	TradeFed instance (or some other harness) running on the HC. For details on
	the host controller, see <a href="/compatibility/vts/host-controller">Host
	Controller Architecture</a>.
	</p>

	<h2 id="resource-providers">Resource providers</h2>

	<p>
	Automated testing requires resources such as system builds, test files, and
	VTS artifacts. While it's possible to build these from source, it is easier to
	build them from tip-of-tree regularly then post the artifacts for download.
	</p>

	<p>
	Partners can access automation resources using the following locations:
	</p>

	<ul>
	<li><strong>Partner Android Build</strong>. Programmatic access granted on a
	per-account basis.</li>
	<li><strong>Local filesystem</strong> (or similar). For partners who do not
	use the Partner Android Build.</li>
	</ul>

	<p>
	For use in flashing the devices later, resources include build providers for
	both options, extending from a single <code>build_provider.py</code> that
	stores the builds in local temporary directories.
	</p>

	<h3 id=partner-android-build>Partner Android Build</h3>

	<p>
	In Android 8.1 and lower releases, Android partners were required to visit the
	Partner Android Build website
	(<a href="https://partner.android.com/build" class="external">https://partner.android.com/build</a>),
	navigate to their account, and fetch the latest system images through the user
	interface. To help partners avoid this slow and labor-intensive process,
	Android {{ androidPVersionNumber }} includes support for automatically
	downloading these resources from PAB when
	the appropriate credentials are provided.
	</p>

	<h4 id="establishing-access">Establishing access</h4>

	<p>
	Programmatic access uses OAuth2 on Google APIs to access the required RPCs.
	Using the
	<a href="https://developers.google.com/api-client-library/python/guide/aaa_oauth#flow_from_clientsecrets" class="external">standard
	approach</a> for generating OAuth2 credentials, the partner must set up a
	client id/secret pair with Google. When the
	<code>PartnerAndroidBuildClient</code> is pointed to that secret for the first
	time, it opens a browser window for the user to log in to their Google
	account, which generates the OAuth2 credentials needed to move forward. The
	credentials (access token and refresh token) are stored locally, meaning
	partners should need to login only once.
	</p>

	<h4>POST request for URL</h4>

	<p>
	Clicking a resource link in PAB sends a POST request that includes the
	necessary data for that resource, including:
	</p>

	<ul>
	<li>build id, build target</li>
	<li>resource name</li>
	<li>branch</li>
	<li>release candidate name and whether or not the candidate is an internal
	build</li>
	</ul>

	<p>
	The POST request is received by the <code>downloadBuildArtifact</code> method
	of the <code>buildsvc</code> RPC, which returns a URL that can be used to
	access the resource.

	<ul>
	<li>For Clockwork Companion APK resources, the URL is a readable URL hosted on
	PAB (which is auth-protected and accessible with the appropriate OAuth2
	credentials).</li>
	<li>For other resources, the URL is long, non-protected URL from the internal
	Android Build API (which expires after five minutes).</li>
	</ul>

	<h4 id="getting-url">Getting the URL </h4>

	<p>
	To avoid cross-site request forgery, the <code>buildsvc</code> RPC requires an
	XSRF token to be POSTed with the other parameters. While this token makes the
	process more secure, it also makes programmatic access much harder since the
	token (which is available only in the JavaScript of the PAB page) is now also
	required for access.
	</p>

	<p>
	To avoid this issue, Android {{ androidPVersionNumber }} redesigns the URL
	naming scheme for all files (not just APKs) to use predictable URL names for
	accessing artifact lists and artifact URLs. The PAB now uses a convenient URL
	format that enables partners to download resources; HC scripts can download
	those APKs easily, since the URL format is known, and HC can bypass the
	XSRF/cookie issues because it does not need the <code>buildsvc</code> RPC.
	</p>

	<h3 id="local-filesystem">Local filesystem</h3>

	<p>
	Given a directory with a list (or zip file) of artifacts, the build provider
	sets the relevant images based on what's in the directory. You can use the
	<a href="https://cloud.google.com/storage/docs/gsutil" class="external">gsutil</a>
	tool to copy files from Google Cloud Storage to a local directory.
	</p>

	<h2 id="flashing-builds">Flashing builds</h2>

	<p>
	After the most recent device images are downloaded to the host, those images
	must be flashed onto the devices. This is done using the standard
	<code>adb</code> and <code>fastboot</code> commands and Python subprocesses,
	based on the temporary file paths stored by the build providers.
	</p>

	<p>
	Supported actions:
	</p>

	<ul>
	<li>Flashing only the GSI</li>
	<li>Flashing individual images from the main system (e.g.,
	<code>fastboot flash boot boot.img</code>)</li>
	<li>Flashing all images from the main system. Example:
	<ul>
	<li><code>fastboot flashall</code> (using the built-in <code>flashall</code>
	utility)</li>
	<li><code>fastboot flash</code> (one at a time)</li>
	</ul>
	</li>
	</ul>

	<h2 id="running=tests">Running tests</h2>

	<p>
	In Android {{ androidPVersionNumber }}, the VTS automated testing
	infrastructure supports only the TradeFed test harness but could be extended
	to support other harnesses in the future.
	</p>

	<p>
	After the devices are prepared, you can invoke tests using one of the
	following options:
	</p>

	<ul>
	<li>When using TradeFed locally, use the <code>test</code> command in the host
	controller, which takes the name of a VTS test plan (e.g.
	<code>vts-selftest</code>) and runs the test.</li>
	<li>When using a TradeFed Cluster (optionally connected to MTT), use the
	<code>lease</code> command in the host controller console, which looks for
	unfulfilled test runs.</li>
	</ul>

	<p>
	If using TradeFedCluster, TradeFed runs
	<a href="/compatibility/vts/host-controller">locally as a remote manager</a>.
	If not, the tests are invoked using Python subprocesses.
	</p>

	<h2 id="reporting-results">Reporting results</h2>

	<p>
	Test results are automatically reported to some VTS dashboard projects by
	<code>VtsMultiDeviceTest</code>.
	</p>

	</body>
	</html>