2023-02-14 Triage Log

Overall a good week for performance with 77 real world crates benchmarks showing an average of nearly 1% performance improvement. Unfortunately, the largest regressions are not yet fully understood and require additional investigation. Of particular interest were some large improvements in doc builds due to storing additional metadata. However, this change might cause some crates to compile slightly slower in incremental check builds, but this is still being investigated.

Triage done by @rylev. Revision range: e4dd9edb..9bb6e60

Summary:

(instructions:u)	mean	range	count
Regressions ❌ (primary)	1.4%	[0.4%, 11.0%]	13
Regressions ❌ (secondary)	0.8%	[0.2%, 1.6%]	4
Improvements ✅ (primary)	-1.4%	[-7.9%, -0.3%]	64
Improvements ✅ (secondary)	-2.1%	[-5.6%, -0.3%]	73
All ❌✅ (primary)	-0.9%	[-7.9%, 11.0%]	77

3 Regressions, 4 Improvements, 9 Mixed; ??? of them in rollups 46 artifact comparisons made in total

Regressions

Rollup of 6 pull requests #107870 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	0.7%	[0.5%, 1.0%]	7
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.7%	[0.5%, 1.0%]	7

Implement deferred_projection_equality for erica solver #107507 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	0.8%	[0.7%, 1.0%]	6
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.8%	[0.7%, 1.0%]	6

rustdoc: Remove cache for preprocessed markdown links #107933 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	1.2%	[1.2%, 1.2%]	1
Regressions ❌ (secondary)	3.9%	[3.9%, 3.9%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	1.2%	[1.2%, 1.2%]	1

Deemed an acceptable trade off for the simplification it brings.

Improvements

ReErased regions are local #107688 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.8%	[-1.2%, -0.3%]	8
Improvements ✅ (secondary)	-1.5%	[-2.6%, -0.4%]	6
All ❌✅ (primary)	-0.8%	[-1.2%, -0.3%]	8

Rollup of 8 pull requests #107811 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	0.6%	[0.6%, 0.6%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.8%	[-1.4%, -0.2%]	7
Improvements ✅ (secondary)	-3.4%	[-4.2%, -1.6%]	7
All ❌✅ (primary)	-0.6%	[-1.4%, 0.6%]	8

Rollup of 9 pull requests #107828 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-1.3%	[-1.4%, -1.2%]	2
Improvements ✅ (secondary)	-3.6%	[-4.1%, -3.2%]	6
All ❌✅ (primary)	-1.3%	[-1.4%, -1.2%]	2

Reduce interning #107869 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-1.4%	[-2.6%, -0.4%]	12
All ❌✅ (primary)	-	-	0

Mixed

Rollup of 8 pull requests #107768 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	1.7%	[1.7%, 1.7%]	2
Improvements ✅ (primary)	-1.0%	[-1.5%, -0.4%]	3
Improvements ✅ (secondary)	-3.7%	[-4.2%, -3.2%]	6
All ❌✅ (primary)	-1.0%	[-1.5%, -0.4%]	3

Small number of regressions in secondary benchmarks in a rollup - I think we're fine calling this triaged.

Update cargo #107778 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	1.1%	[0.4%, 1.5%]	3
Regressions ❌ (secondary)	3.3%	[1.7%, 4.4%]	8
Improvements ✅ (primary)	-2.4%	[-7.7%, -0.4%]	8
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-1.4%	[-7.7%, 1.5%]	11

Most of the regressions are due to noise

Optimize query_cache_hit to reduce code size of the query hot path. #107529 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	0.5%	[0.5%, 0.5%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.3%	[-0.4%, -0.2%]	4
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-0.1%	[-0.4%, 0.5%]	5

Regression is small

Optimize TyKind::eq. #107717 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	1.1%	[1.0%, 1.2%]	2
Regressions ❌ (secondary)	3.6%	[3.3%, 4.2%]	6
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-1.5%	[-1.6%, -1.3%]	2
All ❌✅ (primary)	1.1%	[1.0%, 1.2%]	2

From @nnethercote: keccak and cranelift-codegen are noisy. wg-grammar saw the expected benefit, but it's now considered non-significant, I guess it must have been a bit noisy recently as well.

Resolve documentation links in rustc and store the results in metadata #94857 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	0.6%	[0.3%, 1.5%]	28
Regressions ❌ (secondary)	0.3%	[0.3%, 0.3%]	1
Improvements ✅ (primary)	-2.4%	[-5.7%, -0.6%]	17
Improvements ✅ (secondary)	-3.4%	[-5.2%, -0.6%]	21
All ❌✅ (primary)	-0.6%	[-5.7%, 1.5%]	45

Landed after an MCP and some analysis of the regressions which were deemed acceptable: https://github.com/rust-lang/rust/pull/94857#issuecomment-1414293572

simplify layout calculations in rawvec #107167 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	0.3%	[0.3%, 0.3%]	2
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.4%	[-0.7%, -0.2%]	11
Improvements ✅ (secondary)	-0.6%	[-1.4%, -0.2%]	9
All ❌✅ (primary)	-0.3%	[-0.7%, 0.3%]	13

Improvements greatly exceed regressions here.

Reverse Timsort scan direction #107191 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	0.6%	[0.6%, 0.6%]	1
Regressions ❌ (secondary)	0.2%	[0.2%, 0.2%]	2
Improvements ✅ (primary)	-0.4%	[-0.4%, -0.4%]	2
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-0.1%	[-0.4%, 0.6%]	3

Regressions are small enough that I think we don't need to investigate this closely.

Improve the array::map codegen #107634 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.3%	[0.3%, 0.4%]	6
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.2%	[-0.3%, -0.2%]	4
All ❌✅ (primary)	-	-	0
Perf is a wash

rustc/rustdoc: Perform name resolver cleanups enabled by #94857 #107765 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	13.4%	[13.4%, 13.4%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.4%	[-0.4%, -0.4%]	3
Improvements ✅ (secondary)	-0.2%	[-0.2%, -0.2%]	1
All ❌✅ (primary)	3.1%	[-0.4%, 13.4%]	4

Being investigated by @petrochenkov