Google Git Google Git
Sign in
android/platform/external/ltp/f0df2e76fbeaaeff0e14347055b234c420f5ee65/./android
tree: 885e48f4cdce327e363e82e7c20f64f2814a3ba6 [search]
  1. include/
  2. tools/
  3. Android.bp
  4. AOSP_license_text.txt
  5. empty.sh
  6. how-to-update.txt
  7. ltp_android_walkthrough.md
  8. README.md
  9. skipped-tests.txt
android/README.md

VTS LTP Workflow

A Guide to Developing LTP for VTS/Android

What is LTP?

Github (upstream)

LTP (Linux Test Project) is a suite of tests that covers both kernel interfaces and userspace functionality (glibc, commonly used binaries, etc). For the purposes of Android the userspace functionality testing is of less importance and in fact much of it must be disabled, given the functionality is not available in Android.

As of Jan 2023 there are on the order of 1300 tests executed in VTS on arm64. Most tests are run in both 32-bit and 64-bit mode. Many more are available but currently disabled due to either being broken or not applicable on Android.

Submitting Fixes Upstream

The mailing list for LTP is located here. Some standard kernel guidelines apply to sending patches; they should be checkpatch (scripts/checkpatch.pl in the kernel repository) clean and sent in plain text in canonical patch format. One easy way to do this is by using git format-patch and git send-email.

There is an #LTP channel on freenode. The maintainer Cyril Hrubis is there (his nick is metan).

Landing Fixes in Android

All commits should be pushed to upstream LTP first. We periodically sync from upstream to the Android tree.

If you need to land a fix in the Android tree immediately:

  1. The change must still be submitted upstream first (to the mailing list or as a pull request).
  2. The commit title in the Android tree must start with FROMGIT (if merged upstream) or FROMLIST (if submitted but not yet merged).
  3. The commit message must include a link to the upstream commit or mailing list discussion.

Warning: If you do not follow this process and land code only in the Android tree, your changes are likely to be lost during the next upstream merge/sync.

How is LTP Run in VTS?

The LTP source is located at external/ltp in the Android tree. This is not an exact mirror of upstream, there are outstanding changes to LTP for it to work on Android which have not yet been pushed upstream. In addition to the LTP source there is also the VTS wrapper around it. This is located at test/vts-testcase/kernel/ltp. Some noteworthy directories/files:

  • external/ltp/android/: Contains Android-specific files, aside from Android.[bp, mk] at top level.
  • external/ltp/android/Android.ltp.mk: Lists build rules for the LTP modules built under make. This file gets auto-generated by android/tools/gen_android_build.sh.
  • external/ltp/gen.bp: Lists build rules for the LTP modules built under Soong. This file gets auto-generated by android/tools/gen_android_build.sh.
  • external/ltp/android/ltp_package_list.mk: Lists all tests that will get pulled into VTS - VTS depends on this list. This file gets auto-generated by android/tools/gen_android_build.sh.
  • external/ltp/android/tools/disabled_tests.txt: Lists tests which cannot or should not be compiled for Android. This file is read by gen_android_build.sh during LTP upgrades to produce *.mk files required to build LTP for Android.
  • external/ltp/testcases: Source for LTP tests. Among the most important for the purposes of Treble are those in external/ltp/testcases/kernel/syscalls.
  • test/vts-testcase/kernel/ltp/testcase/tools/configs/disabled_tests.py: Any test listed here will not be run in VTS, despite being compiled.
  • test/vts-testcase/kernel/ltp/testcase/tools/configs/stable_tests.py: Any test listed here will run as part of the vts_ltp_test_arm/vts_ltp_test_arm_64 modules.

Running LTP through atest

You can run LTP tests with atest, which handles all the setup and build steps.

To run all 32 bit LTP tests:

  • atest vts_ltp_test_arm

To run all 64 bit LTP tests:

  • atest vts_ltp_test_arm_64

To run a single test:

  • atest vts_ltp_test_arm:dio.dio13_32bit

Running LTP through VTS

To run VTS LTP it must first be built. VTS is not device specific, you need not compile it specifically for the device you wish to run it on, assuming it is the same architecture.

  • . build/envsetup.sh
  • lunch
  • make -j vts

Then open vts-tradefed and run the VTS stable set:

  • vts-tradefed
  • vts-tf > run vts-kernel -m vts_ltp_test_arm

If you regularly work with multiple devices it may be useful to specify the specific device you wish to run VTS on via the serial number:

  • vts-tf > run vts-kernel -m vts_ltp_test_arm -s 000123456789

Or a specific test within the stable set:

  • vts-tf > run vts-kernel -m vts_ltp_test_arm -t dio.dio13_32bit

Running LTP Directly

Running LTP tests within VTS can be quite cumbersome, especially if you are iterating a lot trying to debug something. Build and run LTP tests faster by doing

  • external/ltp$ mma
  • external/ltp$ adb sync data

The test cases will be located at /data/nativetest{64,}/ltp/testcases/bin.

Sometimes you need to perform this step after syncing:

  • external/ltp$ git clean -x -f -d Otherwise, build will fail.

In order to simulate the exact environment that VTS will be creating for each of these tests, you can set the following environment variables before you run the test. This is very useful if the test itself depends on some of these variables to be set appropriately.

  • adb root && adb shell

In the root shell on device:

  • mkdir -p /data/local/tmp/ltp/tmp/ltptemp
  • mkdir -p /data/local/tmp/ltp/tmp/tmpbase
  • mkdir -p /data/local/tmp/ltp/tmp/tmpdir
  • restorecon -F -R /data/local/tmp/ltp
  • export TMP=/data/local/tmp/ltp/tmp
  • export LTPTMP=/data/local/tmp/ltp/tmp/ltptemp
  • export LTP_DEV_FS_TYPE=ext4
  • export TMPBASE=/data/local/tmp/ltp/tmp/tmpbase
  • export TMPDIR=/data/local/tmp/ltp/tmp/tmpdir
  • export LTPROOT=/data/local/tmp/ltp

For running 64-bit tests:

  • export PATH=/data/nativetest64/ltp/testcases/bin:$PATH

Or For running 32-bit tests:

  • export PATH=/data/nativetest/ltp/testcases/bin:$PATH

How do I enable or disable tests from the LTP build?

Tests are disabled from the LTP build by adding them to external/ltp/android/tools/disabled_tests.txt. Many tests have been added to this file over time. Some of them are not applicable to Android and therefore should not be built. Others were disabled here because they were failing at one point in time for reasons unknown.

To make a change to what is built in LTP, add or remove an entry in this file, and then update the Android-specific build files for LTP, mentioned above:

  • external/ltp/android/Android.ltp.mk, for LTP modules built in make
  • external/ltp/gen.bp, for LTP modules built in soong
  • external/ltp/android/ltp_package_list.mk, which lists all LTP modules that get pulled into VTS

Update these files by running the script located at external/ltp/android/tools/gen_android_build.sh. Instructions for the script are in external/ltp/android/how-to-update.txt.

How do I enable or disable tests from VTS LTP?

In addition to whether modules are built in external/ltp, it is also necessary to configure the VTS harness for LTP to determine which tests are in the stable set, the staging set, or disabled. Note that being disabled in VTS does not affect whether the test is built, but rather determines whether it is run at all as part of the stable or staging sets.

The file test/vts-testcase/kernel/ltp/testcase/tools/configs/stable_tests.py lists tests that will run as part of VTS (vts_ltp_test_arm/vts_ltp_test_arm_64).

When a test is enabled for the first time in VTS it should be in the staging set. The behavior of the test will be observed over a period of time and ensure the test is reliable. After a period of time (a week or two) it will be moved to the stable set.

Tests that will never be relevant to Android should be disabled from the build in external/ltp. Tests that are (hopefully) temporarily broken should be runtime disabled in VTS. The staging and stable sets should normally all be passing. If something is failing there it should either be fixed with priority or disabled until it can be fixed.

If the runtime of LTP changes significantly be sure to update the runtime-hint and test-timeout parameters to VTS in test/vts-testcase/kernel/ltp/stable/AndroidTest.xml.

How do I see recent VTS LTP results?

The internal portal at go/vts11-dashboard shows results for the continuous VTS testing done on internal devices.

Test results are also gathered by Linaro and may be seen here.

Help! The external/ltp build is failing!

Try doing a make distclean inside of external/ltp. If an upgrade to LTP has recently merged or the build files were recently updated, stale files in external/ltp can cause build failures.

What outstanding issues exist?

The hotlist for LTP bugs is ltp-todo.

When you begin working on an LTP bug please assign the bug to yourself so that others know it is being worked on.

Testing x86_64

It is not advisable to run LTP tests directly on your host unless you are fully aware of what the tests will do and are okay with it. These tests may destabilize your box or cause data loss. If you need to run tests on an x86 platform and are unsure if they are safe you should run them in emulation, in a virtualized environment, or on a dedicated development x86 platform.

To run LTP tests for x86 platform, you can do:

  • atest vts_ltp_test_x86
  • atest vts_ltp_test_x86_64

Upgrade LTP source code

While the official LTP releases from upstream are quarterly (see GitHub releases), the external/ltp project in Android is configured to track upstream master branch commits (commit-by-commit) to ensure more regular, often monthly, updates within Android.

The platform/external/ltp project has a tracking branch upstream-master (https://android.googlesource.com/platform/external/ltp/+/refs/heads/upstream-master) that mirrors changes from upstream github.com/linux-test-project/ltp.

Automatic Upgrade (Recommended)

To upgrade LTP to the latest upstream commit and automatically regenerate Android build files, run the following command from your Android checkout root ($ANDROID_BUILD_TOP):

tools/external_updater/updater.sh update external/ltp

This tool will automatically:

  1. Fetch the latest commits from the upstream repository.
  2. Merge the changes into the local branch.
  3. Trigger the post-update hook (post_update.sh) which:
    • Restores Android-specific files and custom runlists during upgrades.
    • Automatically runs the Soong blueprint generator (gen_android_build.sh) to regenerate Android.bp files (requires Docker under the hood; see go/docker for rootless setup).
  4. Create a Git commit with the upgrade.

(Note: If blueprint regeneration is skipped in non-interactive environments due to Docker permissions, run cd android/tools && ./gen_android_build.sh --update manually afterward).

Manual Upgrade

If you need to manually perform the upgrade without using external_updater, follow these steps:

  1. Sync the upstream tracking branch:
    repo sync .
  2. Create a new branch to work on the merge:
    repo start mymerge .
  3. Find the latest commit or release from the upstream tracking branch (goog/upstream-master):
    git log --oneline goog/upstream-master
  4. Merge the upstream commit:
    git merge <upstream commit> --no-ff
  5. Resolve any merge conflicts, verifying the build with:
    git clean -dfx && make autotools && ./configure && make -j
  6. Regenerate Android build targets manually using the helper script:
    git clean -dfx && android/tools/gen_android_build.sh && git clean -dfx && mma .

Manual Verification

After upgrading (via either method), verify the build and run tests:

  1. Build LTP: mma . (from external/ltp).
  2. Run tests: atest vts_ltp_test_arm or atest vts_ltp_test_arm_64.

Writing the Commit Message (Comparing Test Suites)

It is good practice to write a detailed commit message presenting the differences (added/removed tests) in the new LTP version.

You can use the helper script android/tools/compare_ltp_projects.py to compare the tests between the old and new LTP directories:

LTP_NEW=$ANDROID_BUILD_TOP/external/ltp
LTP_OLD=/tmp/ltp-base
mkdir -p $LTP_OLD

# Extract the old version (from main branch) to a temp directory
git archive goog/main | tar -x -C $LTP_OLD

# Compare the two directories
android/tools/compare_ltp_projects.py --ltp-new $LTP_NEW --ltp-old $LTP_OLD