[autotest] only retry tests if they fail with TestFailRetry
For tests that have known flaky failure modes, it is still desirable for
the test to fail (without retries) when an unknown flaky failure occurs.
Therefore, this CL changes the behavior of test retries, so that retires
of a test are only attempted if the test fails with a TestFailRetry
exception. Any other test failure will be treated as a failure
regardless of retries.
This means it is up to the tests in question to catch errors /
exceptions caused by known flaky failure modes, and rethrow those as
TestFailRetry.
For the purposes of testing this change, some changes have been made to
the dummyflake suite.
CQ-DEPEND=Ibaa84f42beac52881cd34351e92474ef1457b15b
BUG=chromium:224372
TEST=added new unit test for test.py
run_remote_tests.sh --remote=<ip> suite:dummyflake;
Suite runs as expected. In particular --
- client/.../dummy_Fail/control.retry_alwaysfail fails immediately even
though the control file uses retries, because the failure is of type
TestFail and hence not retry-able
- The same applies to server/.../dummy_Fail/control.retry_failfast
- client/.../dummy_Fail/control.retry_alwaysflake fails and gets
retried 5 times, failing with a TestRetryFail each time such that it
can be retried
Change-Id: I4d1354cb410856c9de8b720c9f8310cb10d03156
Reviewed-on: https://gerrit.chromium.org/gerrit/46696
Commit-Queue: Aviv Keshet <[email protected]>
Reviewed-by: Aviv Keshet <[email protected]>
Tested-by: Aviv Keshet <[email protected]>
diff --git a/client/common_lib/error.py b/client/common_lib/error.py
index 2432e2b..3a0985f 100644
--- a/client/common_lib/error.py
+++ b/client/common_lib/error.py
@@ -1,3 +1,5 @@
+#pylint: disable-msg=C0111
+
"""
Internal global error types
"""
@@ -223,6 +225,12 @@
exit_status = "WARN"
+class TestFailRetry(TestFail):
+ """Indicates that the test failed, but in a manner that may be retried
+ if test retries are enabled for this test."""
+ exit_status = "FAIL"
+
+
class UnhandledTestError(TestError):
"""Indicates an unhandled error in a test."""
def __init__(self, unhandled_exception):
