update_engine: Implement soft/hard chunk size limits.
The current approach uses a single chunk_size value, that defaults to
-1 (or "whole files") for delta payloads. The "whole files" default is
the preferred value since BSDIFF and SOURCE_BSDIFF operations (the most
common operations in real delta payloads) will benefit from having
access to the whole file in the old and new partition, in particular in
cases like the chrome binary (~120 MiB in size).
On the other hand, very big chunks have a big memory footprint in most
cases. Current implementations of BSDIFF, REPLACE_BZ and REPLACE will
require as much private memory as the destination chunk_size or more.
Because of this and due to the lack of old data, a small chunk_size
(1 MiB) is used for full payloads.
To break this tension between having a big chunk_size for operations
that will benefit from it versus having a small chunk_size for cases
where it doesn't change anything, this patch introduces two chunk
size values: a hard and soft limit.
The hard_chunk_size has the same meaning as the old chunk_size: no
operation should have a destination bigger than the provided hard
limit. The soft_chunk_size is the preferred chunk size for an
operation when a bigger chunk will not benefit significantly the
final payload size. Having a small chunk size for operations like
REPLACE_BZ, REPLACE, SOURCE_COPY and MOVE is important to keep the
memory footprint low when it the extra memory is not required.
The new soft_chunk_size limit is used when merging operations
(previously hard-coded to 1 MiB) and when generating new
operations for zeroed and moved blocks.
BUG=chromium:485397
TEST=Run cros_generate_update_payload for full and delta payloads.
Inspected the chunk size in the resulting payloads.
Change-Id: I370048a81913ad23a151cfef6690627b7fff7277
Reviewed-on: https://chromium-review.googlesource.com/286568
Reviewed-by: Gilad Arnold <[email protected]>
Tested-by: Alex Deymo <[email protected]>
Trybot-Ready: Alex Deymo <[email protected]>
Commit-Queue: Alex Deymo <[email protected]>
16 files changed
