[autotest] In process request/host caching for the rdb.
This cl implements an in process host cache manager for the rdb. The
following considerations were taken into account while designing it:
1. The number of requests outweigh the number of leased hosts
2. The number of net hosts outweighs the number of leased hosts
3. The 'same' request can consult the cache within the span of a single
batched request. These will only be same in terms of host labels/acls
required, not in terms of priority or parent_job_id.
Resulting ramifications:
1. We can't afford to consult the database for each request.
2. We can afford to refresh our in memory representation of a host
just before leasing it.
3. Leasing a host can fail, as we might be using a stale cached host.
4. We can't load a map of all hosts <-> labels each request.
5. Invalidation is hard for most sane, straight-forward choices of
keying hosts against requests.
6. Lower priority requests will starve if they try to lease the same
hosts taken by higher priority requests.
Main design tenets:
1. We can tolerate some staleness in the cache, since we're going
to make sure the host is unleased just before using it.
2. If a job hits a stale cache line it tries again next tick.
3. Trying to invalidate the cache within a single batched request will
be unnecessarily complicated and error prone. Instead, to prevent
starvation, each request only invalidates its cache line, by removing
the hosts it has just leased.
4. The same host may be preset in 2 different cache lines but this won't
matter because each request will check the leased bit in real time before
acquiring it.
5. The entire cache is invalidated at the end of a batched request.
TEST=Ran suites, unittests.
BUG=chromium:366141
DEPLOY=Scheduler
Change-Id: Iafc3ffa876537da628c52260ae692bc2d5d3d063
Reviewed-on: https://chromium-review.googlesource.com/197788
Reviewed-by: Dan Shi <[email protected]>
Tested-by: Prashanth B <[email protected]>
Commit-Queue: Prashanth B <[email protected]>
diff --git a/utils/run_pylint.py b/utils/run_pylint.py
index ef6d9e2..139d8f9 100755
--- a/utils/run_pylint.py
+++ b/utils/run_pylint.py
@@ -62,9 +62,10 @@
# A list of modules for pylint to ignore, specifically, these modules
# are imported for their side-effects and are not meant to be used.
-_IGNORE_MODULES=['common', 'setup_django_environment',
+_IGNORE_MODULES=['common', 'frontend_test_utils',
+ 'setup_django_environment',
'setup_django_lite_environment',
- 'setup_django_readonly_environment', 'setup_test_environment']
+ 'setup_django_readonly_environment', 'setup_test_environment',]
class pylint_error(Exception):
