| <html devsite> |
| <head> |
| <title>Camera HAL Testing Checklist</title> |
| <meta name="project_path" value="/_project.yaml" /> |
| <meta name="book_path" value="/_book.yaml" /> |
| </head> |
| <body> |
| <!-- |
| Copyright 2017 The Android Open Source Project |
| |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| |
| |
| |
| <p>This document lists all tests available for evaluating the Android camera |
| hardware abstraction layer (HAL). It is intended for OEMs and application |
| processor (AP) vendors so they may ensure proper implementation of the camera |
| HAL with minimum defects. Although this is a voluntary addition to the Android |
| Compatibility Test Suite (CTS), it greatly increases camera test coverage and |
| will certainly identify potential bugs.</p> |
| |
| <p>By passing these tests, original equipment manufacturers (OEM) validate |
| whether they have properly integrated the latest Android camera hardware |
| abstraction layer (HAL) 3.2 interfaces. When conforming with all items in the |
| checklist, a device implementation may be considered <em>full</em> with respect |
| to the new Android Camera HAL interfaces. This will in turn enable a device to |
| properly support the new <code>android.hardware.camera2</code> package that |
| camera apps build upon.</p> |
| |
| <h2 id=camera_hal_3_2_specification>Camera HAL 3.2 specification</h2> |
| |
| <p>The Android Camera HAL 3.2 specification is the authoritative source of |
| information on what devices must satisfy; the document here provides a summary |
| of all tests that can be used as a checklist. Camera HAL implementers (e.g. AP |
| vendors) should go through the HAL 3.2 specification line-by-line and ensure |
| their devices conform to it.</p> |
| |
| <p>The current HAL 3.2 specification is defined in these files within the L |
| generic Android Platform Development Kit (PDK):</p> |
| |
| <ul> |
| <li><em>Camera HAL 3.x interface and spec</em>: <code><a |
| href="https://android.googlesource.com/platform/hardware/libhardware/+/master/include/hardware/camera3.h">hardware/libhardware/include/hardware/camera3.h</a></code>, |
| <code><a |
| href="https://android.googlesource.com/platform/hardware/libhardware/+/master/include/hardware/camera_common.h">hardware/libhardware/include/hardware/camera_common.h</a></code> |
| <li><em>Camera HAL 3.x metadata spec</em>: <code><a |
| href="https://android.googlesource.com/platform/system/media/+/master/camera/docs/docs.html">system/media/camera/docs/docs.html</a></code> |
| <li><em>HAL pixel format interface and spec</em>: <code><a |
| href="https://android.googlesource.com/platform/system/core/+/master/include/system/graphics.h">system/core/include/system/graphics.h</a></code> |
| </ul> |
| |
| <h2 id=camera_test_types>Camera test types</h2> |
| |
| <p>Here are the primary types of tests available for the latest Android camera |
| along with references to associated instructions below:</p> |
| |
| <ul> |
| <li><em><a href="#vendor_test_suite">Vendor Test Suite (VTS)</a>:</em> Tests that directly test the |
| camera HAL interface |
| <li><em><a href="#cts_tests">Compatibility Test Suite (CTS)</a></em>: |
| Standard, automated Android |
| tests to ensure device compatibility - see the |
| <a href="/compatibility/cts/index.html">CTS introduction</a> and the <a |
| href="/devices/tech/test_infra/tradefed/index.html">Trade Federation |
| Overview</a> |
| <li><em><a href="#its_tests">Image Test Suite (ITS)</a>:</em> Manually run |
| tests to ensure image correctness - see the top-level and test-specific |
| <code>README</code> files and <code>tutorial.py</code> for setup instructions |
| <li><em><a |
| href="#manual_testingcam_tests">Manual TestingCam tests</a>:</em> |
| Run from the source in <code>pdk/apps/TestingCamera/</code> |
| <li><em><a href="#manual_testingcam2_tests">Manual TestingCam2.1 |
| tests</a>:</em> Run from the source in <code>pdk/apps/TestingCamera2/</code> |
| </ul> |
| |
| <p>All of these test types are described in detail below. These tests are |
| presented in the chronological order in which OEMs are expected to execute |
| them.</p> |
| |
| <p>For instance, if a device fails the native tests, it will assuredly fail the |
| subsequent Compatibility Test Suite (CTS) tests. And if a device fails CTS, |
| there is little use in proceeding to the Image Test Suite (ITS). We recommend |
| addressing failures in each test type before proceeding to the next set of |
| tests.</p> |
| |
| <h2 id=vendor_test_suite>Vendor Test Suite (VTS) tests</h2> |
| |
| <p>The Android Vendor Test Suite (VTS) is a testing suite that works on the |
| HIDL interface level. For more information on using VTS, see |
| <a href="/compatibility/vts/">Vendor Test Suite</a>.</p> |
| |
| <h2 id=cts_tests>Compatibility Test Suite (CTS) tests</h2> |
| |
| <p>Camera Android Compatibility Test Suite (CTS) tests focus upon device |
| compatibility. They do not require a specific test environment (the field of |
| view or FOV CTS Verifier test being the lone exception). </p> |
| |
| <p>The starting path for Camera CTS tests is: <code>platform/cts</code>.</p> |
| |
| <p>When running Camera CTS for devices that support external cameras (such as |
| USB webcams), you must have a device plugged in when running CTS or the tests |
| will automatically fail.</p> |
| |
| <p>See the <a href="/compatibility/cts/index.html">CTS |
| introduction</a> and its subpages for general instructions on running CTS.</p> |
| |
| <h3 id=cts_tests_for_the_android_hardware_camera_api>CTS tests for |
| the <code>android.hardware.Camera</code> API</h3> |
| |
| <p>Find these camera tests under <code>cts/tests/tests/</code>:</p> |
| |
| <ul> |
| <li><code>hardware/src/android/hardware/cts/CameraTest.java</code> |
| <li><code>hardware/src/android/hardware/cts/CameraGLTest.java</code> |
| <li><code>hardware/src/android/hardware/cts/Camera_SizeTest.java</code> |
| <li><code>permission/src/android/permission/cts/CameraPermissionTest.java</code> |
| </ul> |
| |
| <h3 id=cts_tests_for_the_android_hardware_camera2_api>CTS tests for |
| the <code>android.hardware.camera2</code> API</h3> |
| |
| <p>Find these camera tests under <code>cts/tests/tests/</code>:</p> |
| |
| <ul> |
| <li><code>hardware/src/android/hardware/camera2/cts/*</code> |
| <li><code>permission/src/android/permission/cts/Camera2PermissionTest.java</code> |
| </ul> |
| |
| <h3 id=cts_verifier_camera_tests>CTS Verifier camera tests</h3> |
| |
| <p>Find these camera tests under: |
| <code>cts/apps/CtsVerifier/src/com/android/cts/verifier/camera/*</code></p> |
| |
| <h2 id=its_tests>Image Test Suite (ITS) tests</h2> |
| |
| <p>The CameraITS tests focus upon image correctness. These Python scripts are |
| manually run on a workstation with the Android device connected over USB. The |
| workstation can run any operating system as long as it has the requisite Python |
| 2.7 environment.</p> |
| |
| <p class="note">Since ITS is a CTS Verifier subtest, start CTS Verifier and the |
| ITS subtest before running the python scripts so they have processes with which |
| to communicate.</p> |
| |
| <p>The CameraITS infrastructure and tests are located under: |
| <code>cts/apps/CameraITS</code></p> |
| |
| <p>See the latest <code>README</code> file in this top-level folder for |
| instructions on how to set up and run the tests. For setup: <code>make |
| cts</code></p> |
| |
| <pre class="devsite-click-to-copy"> |
| <code class="devsite-terminal">extract root/out/host/linux-x86/cts-verfier/android-cts-verifier.zip</code> |
| <code class="devsite-terminal">cd android-cts-verifier</code> |
| <code class="devsite-terminal">adb install -r CtsVerifier.apk</code> |
| <code class="devsite-terminal">cd CameraITS</code> |
| </pre> |
| |
| <p>See <code>tutorial.py</code> in the <code>tests</code> subdirectory for a |
| walkthrough of the script's use. Each test resides in a corresponding |
| <code>tests/scene<#></code> subdirectory. See the <code>README</code> file in |
| each subdirectory for specific test instructions.</p> |
| |
| <p>To follow the recommended way of setting up and running the Camera Image Test |
| Suite, see <a href="/compatibility/cts/camera-its-box">Camera |
| ITS-in-a-Box</a>.</p> |
| |
| <p>To run ITS manually, you will need a simple physical environment with a |
| specific, reusable target |
| such as a white wall, grey card, and desk lamp. The Android device is mounted |
| on a tripod and its camera functions are exercised by the scripts. Most tests |
| are pass or fail but some offer metrics, as well.</p> |
| |
| <p>These tests are works-in-progress and are not yet comprehensive enough for |
| full automated pass/fail validation of the camera HAL. However, these scripts |
| do test scenarios that are not tested in CTS and are an important component of |
| the overall HAL 3.2 test plan.</p> |
| |
| <h3 id=its_tests_on_scene_0_plain>ITS tests on scene 0 (plain)</h3> |
| |
| <p>This test requires no specific setup. Pass all of the tests in the |
| <code>tests/scene0</code> folder, for all cameras (back + front + any |
| others).</p> |
| |
| <h3 id=its_tests_on_scene_1_grey_card>ITS tests on scene 1 (grey card)</h3> |
| |
| <p>Pass all of the tests in the <code>tests/scene1</code> folder, for all |
| cameras (back + front + any others). The <code>tests/scene1/README</code> file |
| describes the scene setup.</p> |
| |
| <h3 id=its_tests_on_scene_2_camera_lab>ITS tests on scene 2 (camera lab)</h3> |
| |
| <p>Pass all of the tests in the <code>tests/scene2</code> folder, for all |
| cameras (back + front + any others). The <code>tests/scene2/README</code> file |
| describes the scene setup.</p> |
| |
| <h2 id=media_framework_tests>Media Framework tests</h2> |
| |
| <p>Pass all of the camera-related media tests in MediaFrameworkTest. Please |
| note, these tests require the mediaframeworktest.apk be installed on the Android |
| device. You will need to <code>make mediaframeworktest</code> and then use adb |
| to install the resulting .apk. Example commands are included below.</p> |
| |
| <p>The starting path for Camera-related media framework tests is: |
| <code>platform/frameworks/base</code></p> |
| |
| <p>Find the source code for the tests here: |
| <code>frameworks/base/media/tests/MediaFrameworkTest</code></p> |
| |
| <p>To set up these tests:</p> |
| |
| <pre class="devsite-click-to-copy"> |
| <code class="devsite-terminal">make mediaframeworktest</code> |
| <code class="devsite-terminal">adb install out/target/product/<em><name></em>/data/app/mediaframeworktest.apk</code> |
| </pre> |
| |
| <p>Where the <em><code><name></code></em> variable represents the directory |
| containing the vendor's product.</p> |
| |
| <p>Find all of the tests in the following directory or its subdirectories:</p> |
| |
| <pre class="devsite-click-to-copy"> |
| frameworks/base/media/tests/MediaFrameworkTest/src/com/android/mediaframeworktest |
| </pre> |
| |
| <p>Each subdirectory represents a class of tests:</p> |
| |
| <ul> |
| <li><code>functional/</code> |
| <li><code>integration/</code> |
| <li><code>performance/</code> |
| <li><code>power/</code> |
| <li><code>stress/</code> |
| <li><code>unit/</code> |
| </ul> |
| |
| <h3 id=running_media_framework_tests>Running Media Framework tests</h3> |
| |
| <p>To see all of the available tests::</p> |
| |
| <pre class="devsite-terminal devsite-click-to-copy"> |
| adb shell pm list instrumentation |
| </pre> |
| |
| <p>This will yield results resembling:</p> |
| |
| <pre class="devsite-click-to-copy"> |
| instrumentation:com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner |
| (target=com.android.mediaframeworktest) |
| instrumentation:com.android.mediaframeworktest/.MediaRecorderStressTestRunner |
| (target=com.android.mediaframeworktest) |
| instrumentation:com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner |
| (target=com.android.mediaframeworktest) |
| instrumentation:com.android.mediaframeworktest/.MediaFrameworkPowerTestRunner |
| (target=com.android.mediaframeworktest) |
| </pre> |
| |
| <p>Identify and extract the component (between <code>instrumentation:</code> |
| and <code>(target=com.android.mediaframeworktest) </code>from each test line. |
| The component is composed of the target package name |
| (<code>com.android.mediaframeworktest</code>) and the test runner name |
| (<code>MediaFramework<type>TestRunner</code>).</p> |
| |
| <p>For instance:</p> |
| |
| <pre class="devsite-click-to-copy"> |
| com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner |
| com.android.mediaframeworktest/.MediaRecorderStressTestRunner |
| com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner |
| com.android.mediaframeworktest/.MediaFrameworkPowerTestRunner |
| </pre> |
| |
| <p>You may then pass each component to <code>adb shell am instrument</code> like |
| so:</p> |
| |
| <pre class="devsite-terminal devsite-click-to-copy"> |
| adb shell am instrument -w <var>component.name</var> |
| </pre> |
| |
| <p>Where <code><var>component.name</var></code> equals the extracted value |
| above. For |
| example:</p> |
| |
| <pre class="devsite-terminal devsite-click-to-copy"> |
| adb shell am instrument -w com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner |
| </pre> |
| |
| <p>Please note, while the class path is the Java package + class name, the |
| instrumentation package isn't necessarily the same as the Java package. Make |
| sure you use the AndroidManifest.xml package when concatenating the component |
| name, not the Java package in which the test runner class resides.</p> |
| |
| <p>To run a single class of tests, pass the -e class <test-class> argument, like |
| so:</p> |
| |
| <pre class="devsite-terminal devsite-click-to-copy"> |
| adb shell am instrument -e class com.android.mediaframeworktest.integration.CameraBinderTest -w com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner |
| </pre> |
| |
| <p>To run only a single method in a test class, append a pound (#) sign and the |
| method name (in this case, <code>testConnectPro</code>) to the class name, like so:</p> |
| |
| <pre class="devsite-terminal devsite-click-to-copy"> |
| adb shell am instrument -e class 'com.android.mediaframeworktest.integration.CameraBinderTest#testConnectPro' -w com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner |
| </pre> |
| |
| <h3 id=media_settings_functional_tests>Media settings functional tests</h3> |
| |
| <p>Here is an example run of a functional test. This test verifies the basic |
| functionality of different combinations of camera settings. (ie, Flash, |
| exposure, WB, scene, picture size and geoTag)</p> |
| |
| <p>Run the test command:</p> |
| <pre class="devsite-terminal devsite-click-to-copy"> |
| adb shell am instrument -w -r -e delay_msec 15 -e log true -e class com.android.mediaframeworktest.functional.camera.CameraPairwiseTest com.android.mediaframeworktest/com.android.mediaframeworktest.CameraStressTestRunner |
| </pre> |
| |
| <h3 id=media_integration_tests>Media integration tests</h3> |
| |
| <p>Here is an example run of an integration test, in this case |
| mediaframeworktest/integration/CameraBinderTest.java and |
| mediaframeworktest/CameraStressTestRunner.java:</p> |
| |
| <pre class="devsite-terminal devsite-click-to-copy"> |
| adb shell am instrument -e class \ 'com.android.mediaframeworktest.<strong>integration</strong>.<strong>CameraBinderTest'</strong> -w \ 'com.android.mediaframeworktest/.<strong>CameraStressTestRunner'</strong> |
| </pre> |
| |
| <p>If successful, this results in output resembling:</p> |
| |
| <pre class="devsite-click-to-copy"> |
| ----- |
| |
| com.android.mediaframeworktest.integration.CameraBinderTest:........... |
| Test results for CameraStressTestRunner=........... |
| Time: 3.328 |
| |
| OK (11 tests) |
| |
| ----- |
| </pre> |
| |
| <h3 id=media_performance_tests>Media performance tests</h3> |
| |
| <p>This preview memory test will open and release the camera preview for 200 |
| times. In each 20 iterations, the snapshot of ps mediaserver will be recorded |
| and it will compare the memory usage different after 200 iterations. Test will |
| fail If the difference is greater than 150kM.</p> |
| |
| <p>Run the test command:</p> |
| <pre class="devsite-terminal devsite-click-to-copy"> |
| adb shell am instrument -w -r -e class com.android.mediaframeworktest.performance.MediaPlayerPerformance#testCameraPreviewMemoryUsage com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner |
| </pre> |
| |
| <p>More detailed output can be found in: |
| <code>/sdcard/mediaMemOutput.txt</code></p> |
| |
| <h3 id=media_unit_tests>Media unit tests</h3> |
| |
| <p>The commands to run unit tests are all similar. For example, for |
| CameraMetadataTest.java, the command would be:</p> |
| |
| <pre class="devsite-terminal devsite-click-to-copy"> |
| adb shell am instrument -e class 'com.android.mediaframeworktest.unit.CameraMetadataTest' -w 'com.android.mediaframeworktest/.CameraStressTestRunner' |
| </pre> |
| |
| <h3 id=media_stress_tests>Media stress tests</h3> |
| |
| <p>This test is to stress out the camera image capture and video recording.</p> |
| |
| <p>Run the test command:</p> |
| |
| <pre class="devsite-terminal devsite-click-to-copy"> |
| adb shell am instrument -w com.google.android.camera.tests/com.android.camera.stress.CameraStressTestRunner |
| </pre> |
| |
| <p>All tests should pass.</p> |
| |
| <h2 id=manual_testingcam_tests>Manual TestingCam tests</h2> |
| |
| <p>The TestingCam app should be run manually with the following checks performed. |
| The source for TestingCam is here: <code>pdk/apps/TestingCamera/</code></p> |
| |
| <h3 id=infinity_focus_with_camera_tilt>Infinity focus with camera tilt</h3> |
| |
| <p>Start TestingCam, turn on preview, and ensure that autofocus mode is set to |
| infinity. Using the <strong>Take picture</strong> button, capture shots of |
| distant subjects (at least 10m away) with the camera pointed horizontally, |
| upwards (close to vertical), and downwards (close to vertical); an example of |
| the upwards shot could be high leaves/branches of a tree from beneath and an |
| example of the downwards shot could be the street as seen from the roof of a |
| building. In all cases, the distant subject should be sharp and in focus. Save |
| and view the shots in the gallery view so that you can zoom in and inspect the |
| sharpness more easily.</p> |
| |
| <p>Note that for a camera with a VCM actuator to pass this test, it will require |
| either a closed-loop AF control system, or it will need some sort of SW |
| correction based on using accelerometer data to determine camera orientation. |
| Reliable factory calibration of the lens infinity position will also be needed.</p> |
| |
| <h2 id=manual_testingcam2_tests>Manual TestingCam2 tests</h2> |
| |
| <p>The TestingCam2 app should be run manually, with the following checks |
| performed. The source for TestingCam2 is here: <code>pdk/apps/TestingCamera2/</code></p> |
| |
| <h3 id=9_1_jpeg_capture>JPEG capture</h3> |
| |
| <p>Start TestingCam2, and press the <strong>JPEG</strong> button. The image |
| that appears to the right of the viewfinder image should appear the same as the |
| viewfinder, including having the same orientation.</p> |
| |
| </body> |
| </html> |
