en/compatibility/vts/ui.html - platform/docs/source.android.com - Git at Google

 <html devsite>
   <head>
     <title>VTS Dashboard UI</title>
     <meta name="project_path" value="/_project.yaml" />
     <meta name="book_path" value="/_book.yaml" />
   </head>
   <body>
   <!--
       Copyright 2017 The Android Open Source Project

       Licensed under the Apache License, Version 2.0 (the "License");
       you may not use this file except in compliance with the License.
       You may obtain a copy of the License at

           http://www.apache.org/licenses/LICENSE-2.0

       Unless required by applicable law or agreed to in writing, software
       distributed under the License is distributed on an "AS IS" BASIS,
       WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
       See the License for the specific language governing permissions and
       limitations under the License.
   -->


 <p>
 The VTS Dashboard provides a cohesive user interface that uses material design
 to effectively display information about test results, profiling, and coverage.
 Dashboard styling uses open-source Javascript libraries including Materialize
 CSS and jQueryUI to process data delivered by Java servlets in Google App
 Engine.
 </p>

 <h2>Dashboard home</h2>
 <p>
 The Dashboard home page displays a list of test suites a user has added to
 favorites.
 </p>
 <img src="images/treble_vts_ui_main.png" title="VTS Dashboard landing page">
 <figcaption><strong>Figure 1.</strong> VTS Dashboard, home page.</figcaption>

 <p>
 From this list, users can:
 </p>
 <ul>
 <li>Select a test suite to view results for that suite.
 <li>Click <strong>SHOW ALL</strong> to view all VTS test names.
 <li>Select the <strong>Edit</strong> icon to modify the Favorites list.<br>
 <img src="images/treble_vts_ui_favorites.png" title="VTS Dashboard favorites">
 <figcaption><strong>Figure 2.</strong> VTS Dashboard, editing Favorites
 page.</figcaption></li>
 </ul>

 <h2 id=test-results>Test results</h2>
 <p>
 Test Results displays the latest information about the selected test suite,
 including a list of profiling points, a table of test case results in
 chronological order, and a pie chart displaying the result breakdown of the
 latest run (users can load older data by paging right).
 </p>

 <img src="images/treble_vts_ui_results.png" title="VTS Dashboard results">
 <figcaption><strong>Figure 3.</strong> VTS Dashboard, test results.</figcaption>

 <p>
 Users can filter data using queries or by modifying the test type (pre-submit,
 post-submit, or both). Search queries support general tokens and field-specific
 qualifiers; supported search fields are: device build ID, branch, target name,
 device name, and test build ID. These are specified in the format:
 <var>FIELD-ID</var>="<var>SEARCH QUERY</var>". Quotes are used to treat multiple
 words as a single token to match with the data in the columns.
 </p>

 <h2 id=profiling>Data profiling</h2>
 <p>
 Users can select a profiling point to reach an interactive view of the
 quantitative data for that point in a <strong>line graph</strong> or
 <strong>histogram</strong> (examples below). By default, the view displays the
 latest information; users can use the date picker to load specific time windows.
 </p>
 <img src="images/treble_vts_ui_performance.png" title="VTS Dashboard performance">
 <figcaption><strong>Figure 4.</strong> VTS Dashboard, line graph performance.
 </figcaption>
 <p>
 Line graphs display data from a collection of unordered performance values,
 which can be useful when a test of performance produces a vector of performance
 values that vary as a function of another variable (e.g., throughput versus
 message size).
 </p>
 <img src="images/treble_vts_ui_histogram.png" title="VTS Dashboard histogram">
 <figcaption><strong>Figure 5.</strong> VTS Dashboard, histogram performance.</figcaption>

 <h2 id=coverage>Test coverage</h2>
 <p>
 Users can view coverage information from the coverage percent link in test
 results.
 </p>
 <img src="images/treble_vts_ui_coverage.png" title="VTS Dashboard coverage">
 <figcaption>
 <strong>Figure 6.</strong> VTS Dashboard, coverage percentages.</figcaption>

 <p>
 For each test case and source file, users can view an expandable element
 containing color-coded source code according to the coverage provided by the
 selected test:
 </p>
 <img src="images/treble_vts_ui_coverage_source.png" title="VTS Dashboard coverage_source">
 <figcaption>
 <strong>Figure 7.</strong> VTS Dashboard, coverage source code.</figcaption>

 <ul>
 <li>Uncovered lines are highlighted <font style="color:red">red</font>.</li>
 <li>Covered lines are highlighted <font style="color:green">green</font>.</li>
 <li>Non-executable lines are <strong>uncolored</strong>.</li>
 </ul>

 <p>
 Coverage information is grouped depending into sections depending on how it was
 provided at run-time. Tests may upload coverage:
 </p>
 <ul>
 <li><strong>Per function</strong>. Section headers have the format "Coverage:
 <var>FUNCTION-NAME</var>".</li>
 <li><strong>In Total</strong> (provided at the end of the test run). Only one
 header is present: "Coverage: All".</li>
 </ul>

 <p>
 The Dashboard fetches source code client-side from a server, which uses the
 open-source
 <a href="https://gerrit-review.googlesource.com/Documentation/rest-api.html"
 class="external">Gerrit REST API</a>.
 </p>

 <h2 id=monitor>Monitoring &amp; testing</h2>
 <p>
 The VTS Dashboard provides the following monitors and unit tests.
 </p>
 <ul>
 <li><strong>Test email alerts</strong>. Alerts are configured in a Cron job that
 executes at a fixed interval of two (2) minutes. The job reads the VTS status
 table to determine if new data has been uploaded to each table, done by checking
 the test's raw data upload timestamp is newer than the last status update
 timestamp. If the upload timestamp is newer, the job queries for new data
 between now and the last raw data upload. New test case failures, continued test
 case failures, transient test case failures, test case fixes, an inactive tests
 are determined; this information is then sent in email format to the subscribers
 of each test.</li>
 <li><strong>Web service health</strong>. Google Stackdriver integrates with
 Google App Engine to provide easy monitoring of the VTS Dashboard. Simple uptime
 checks verify pages can be accessed while other tests can be created to verify
 latency on each page, servlet, or database. These checks ensure the Dashboard is
 always accessible (else an administrator will be notified).</li>
 <li><strong>Analytics</strong>. You can integrate a VTS Dashboard page with
 Google Cloud Analytics by specifying a valid Analytics ID in the page
 configuration (the pom.xml file). Integration provides a more robust analysis of
 page usage, user interaction, locality, session statistics, etc.</li>
 </ul>

   </body>
 </html>
	<html devsite>
	<head>
	<title>VTS Dashboard UI</title>
	<meta name="project_path" value="/_project.yaml" />
	<meta name="book_path" value="/_book.yaml" />
	</head>
	<body>
	<!--
	Copyright 2017 The Android Open Source Project

	Licensed under the Apache License, Version 2.0 (the "License");
	you may not use this file except in compliance with the License.
	You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing, software
	distributed under the License is distributed on an "AS IS" BASIS,
	WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	See the License for the specific language governing permissions and
	limitations under the License.
	-->


	<p>
	The VTS Dashboard provides a cohesive user interface that uses material design
	to effectively display information about test results, profiling, and coverage.
	Dashboard styling uses open-source Javascript libraries including Materialize
	CSS and jQueryUI to process data delivered by Java servlets in Google App
	Engine.
	</p>

	<h2>Dashboard home</h2>
	<p>
	The Dashboard home page displays a list of test suites a user has added to
	favorites.
	</p>
	<img src="images/treble_vts_ui_main.png" title="VTS Dashboard landing page">
	<figcaption><strong>Figure 1.</strong> VTS Dashboard, home page.</figcaption>

	<p>
	From this list, users can:
	</p>
	<ul>
	<li>Select a test suite to view results for that suite.
	<li>Click <strong>SHOW ALL</strong> to view all VTS test names.
	<li>Select the <strong>Edit</strong> icon to modify the Favorites list.<br>
	<img src="images/treble_vts_ui_favorites.png" title="VTS Dashboard favorites">
	<figcaption><strong>Figure 2.</strong> VTS Dashboard, editing Favorites
	page.</figcaption></li>
	</ul>

	<h2 id=test-results>Test results</h2>
	<p>
	Test Results displays the latest information about the selected test suite,
	including a list of profiling points, a table of test case results in
	chronological order, and a pie chart displaying the result breakdown of the
	latest run (users can load older data by paging right).
	</p>

	<img src="images/treble_vts_ui_results.png" title="VTS Dashboard results">
	<figcaption><strong>Figure 3.</strong> VTS Dashboard, test results.</figcaption>

	<p>
	Users can filter data using queries or by modifying the test type (pre-submit,
	post-submit, or both). Search queries support general tokens and field-specific
	qualifiers; supported search fields are: device build ID, branch, target name,
	device name, and test build ID. These are specified in the format:
	<var>FIELD-ID</var>="<var>SEARCH QUERY</var>". Quotes are used to treat multiple
	words as a single token to match with the data in the columns.
	</p>

	<h2 id=profiling>Data profiling</h2>
	<p>
	Users can select a profiling point to reach an interactive view of the
	quantitative data for that point in a <strong>line graph</strong> or
	<strong>histogram</strong> (examples below). By default, the view displays the
	latest information; users can use the date picker to load specific time windows.
	</p>
	<img src="images/treble_vts_ui_performance.png" title="VTS Dashboard performance">
	<figcaption><strong>Figure 4.</strong> VTS Dashboard, line graph performance.
	</figcaption>
	<p>
	Line graphs display data from a collection of unordered performance values,
	which can be useful when a test of performance produces a vector of performance
	values that vary as a function of another variable (e.g., throughput versus
	message size).
	</p>
	<img src="images/treble_vts_ui_histogram.png" title="VTS Dashboard histogram">
	<figcaption><strong>Figure 5.</strong> VTS Dashboard, histogram performance.</figcaption>

	<h2 id=coverage>Test coverage</h2>
	<p>
	Users can view coverage information from the coverage percent link in test
	results.
	</p>
	<img src="images/treble_vts_ui_coverage.png" title="VTS Dashboard coverage">
	<figcaption>
	<strong>Figure 6.</strong> VTS Dashboard, coverage percentages.</figcaption>

	<p>
	For each test case and source file, users can view an expandable element
	containing color-coded source code according to the coverage provided by the
	selected test:
	</p>
	<img src="images/treble_vts_ui_coverage_source.png" title="VTS Dashboard coverage_source">
	<figcaption>
	<strong>Figure 7.</strong> VTS Dashboard, coverage source code.</figcaption>

	<ul>
	<li>Uncovered lines are highlighted <font style="color:red">red</font>.</li>
	<li>Covered lines are highlighted <font style="color:green">green</font>.</li>
	<li>Non-executable lines are <strong>uncolored</strong>.</li>
	</ul>

	<p>
	Coverage information is grouped depending into sections depending on how it was
	provided at run-time. Tests may upload coverage:
	</p>
	<ul>
	<li><strong>Per function</strong>. Section headers have the format "Coverage:
	<var>FUNCTION-NAME</var>".</li>
	<li><strong>In Total</strong> (provided at the end of the test run). Only one
	header is present: "Coverage: All".</li>
	</ul>

	<p>
	The Dashboard fetches source code client-side from a server, which uses the
	open-source
	<a href="https://gerrit-review.googlesource.com/Documentation/rest-api.html"
	class="external">Gerrit REST API</a>.
	</p>

	<h2 id=monitor>Monitoring & testing</h2>
	<p>
	The VTS Dashboard provides the following monitors and unit tests.
	</p>
	<ul>
	<li><strong>Test email alerts</strong>. Alerts are configured in a Cron job that
	executes at a fixed interval of two (2) minutes. The job reads the VTS status
	table to determine if new data has been uploaded to each table, done by checking
	the test's raw data upload timestamp is newer than the last status update
	timestamp. If the upload timestamp is newer, the job queries for new data
	between now and the last raw data upload. New test case failures, continued test
	case failures, transient test case failures, test case fixes, an inactive tests
	are determined; this information is then sent in email format to the subscribers
	of each test.</li>
	<li><strong>Web service health</strong>. Google Stackdriver integrates with
	Google App Engine to provide easy monitoring of the VTS Dashboard. Simple uptime
	checks verify pages can be accessed while other tests can be created to verify
	latency on each page, servlet, or database. These checks ensure the Dashboard is
	always accessible (else an administrator will be notified).</li>
	<li><strong>Analytics</strong>. You can integrate a VTS Dashboard page with
	Google Cloud Analytics by specifying a valid Analytics ID in the page
	configuration (the pom.xml file). Integration provides a more robust analysis of
	page usage, user interaction, locality, session statistics, etc.</li>
	</ul>

	</body>
	</html>