2023-03-07 Triage Log

A really quiet week with almost all regressions being due to noise in benchmarks that show “bimodality” in codegen that can cause swings in performance from one change to the other. The only true performance change was a two-line change by @nnethercote to remove a redundant function call which led to a 0.3% improvement in performance across roughly 15 benchmarks.

Triage done by @rylev. Revision range: 31f858d9..8f9e09ac

Summary:

(instructions:u)	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.0%	[1.2%, 2.8%]	8
Improvements ✅ (primary)	-0.4%	[-0.7%, -0.2%]	7
Improvements ✅ (secondary)	-1.0%	[-1.8%, -0.1%]	31
All ❌✅ (primary)	-0.4%	[-0.7%, -0.2%]	7

7 Regressions, 8 Improvements, 2 Mixed; 7 of them in rollups 35 artifact comparisons made in total

Regressions

Rollup of 5 pull requests #108640 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.2%	[2.1%, 4.4%]	8
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

• @nnethercote expects that deep-vector may have become susceptible to bimodal codegen which can cause swings in perf performance.

Revert stabilization of #![feature(target_feature_11)] #108654 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	0.7%	[0.6%, 0.7%]	3
Regressions ❌ (secondary)	0.4%	[0.3%, 0.5%]	4
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.7%	[0.6%, 0.7%]	3

• Noise: small regressions are immediately reversed in the next run.

Add vectored positioned I/O on Unix #89518 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	0.2%	[0.2%, 0.3%]	4
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.2%	[0.2%, 0.3%]	4

• Small regression in Diesel which has started to show some bimodality. I think it's safe to call this triaged.

Remove NormalizationError::ConstantKind #108689 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	1.1%	[1.0%, 1.1%]	2
Regressions ❌ (secondary)	2.3%	[2.1%, 2.7%]	6
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.6%	[-0.6%, -0.6%]	1
All ❌✅ (primary)	1.1%	[1.0%, 1.1%]	2

• more keccak and cranelift-codegen noise.

Bump version to 1.70.0 #108740 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	0.3%	[0.3%, 0.4%]	6
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.3%	[0.3%, 0.4%]	6

• Perhaps due to shuffled hashes? In any case, not much that can be done.

Add more debug assertions to unsafe functions #105117 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	1.0%	[0.9%, 1.1%]	2
Regressions ❌ (secondary)	2.3%	[2.0%, 2.7%]	6
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	1.0%	[0.9%, 1.1%]	2

• more cranelift-codegen noise.

Rollup of 7 pull requests #108789 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	1.1%	[1.0%, 1.1%]	2
Regressions ❌ (secondary)	2.3%	[2.1%, 2.7%]	6
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	1.1%	[1.0%, 1.1%]	2

• more keccak and cranelift-codegen noise.

Improvements

Remove or document uses of #[rustc_box] in library #108476 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.6%	[-0.7%, -0.5%]	4
Improvements ✅ (secondary)	-0.3%	[-0.3%, -0.3%]	3
All ❌✅ (primary)	-0.6%	[-0.7%, -0.5%]	4

• Revert of noise

Rollup of 8 pull requests #108677 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.6%	[-0.7%, -0.6%]	3
Improvements ✅ (secondary)	-0.4%	[-0.5%, -0.3%]	4
All ❌✅ (primary)	-0.6%	[-0.7%, -0.6%]	3

• Revert of noise

Rollup of 8 pull requests #108709 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-1.0%	[-1.1%, -0.9%]	2
Improvements ✅ (secondary)	-2.3%	[-2.6%, -2.0%]	6
All ❌✅ (primary)	-1.0%	[-1.1%, -0.9%]	2

• Revert of noise

Rollup of 6 pull requests #108732 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.3%	[-0.4%, -0.3%]	6
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-0.3%	[-0.4%, -0.3%]	6

• Revert of noise

Don't call temporary_scope twice. #108692 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.3%	[-0.6%, -0.2%]	15
Improvements ✅ (secondary)	-1.0%	[-1.5%, -0.4%]	27
All ❌✅ (primary)	-0.3%	[-0.6%, -0.2%]	15

Desugaring of drop and replace at MIR build #107844 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-1.0%	[-1.1%, -1.0%]	2
Improvements ✅ (secondary)	-2.0%	[-2.6%, -0.3%]	7
All ❌✅ (primary)	-1.0%	[-1.1%, -1.0%]	2

Use partial_cmp to implement tuple lt/le/ge/gt #108157 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-1.2%	[-1.6%, -0.6%]	3
Improvements ✅ (secondary)	-2.2%	[-2.6%, -2.0%]	6
All ❌✅ (primary)	-1.2%	[-1.6%, -0.6%]	3

Erase regions even when failing to normalize type in MIR opts #108787 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-1.0%	[-1.1%, -1.0%]	2
Improvements ✅ (secondary)	-2.2%	[-2.6%, -2.0%]	6
All ❌✅ (primary)	-1.0%	[-1.1%, -1.0%]	2

Mixed

Rollup of 10 pull requests #108587 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	0.7%	[0.7%, 0.8%]	4
Regressions ❌ (secondary)	0.4%	[0.3%, 0.6%]	4
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.8%	[-0.8%, -0.8%]	1
All ❌✅ (primary)	0.7%	[0.7%, 0.8%]	4

• Noise that is counter balanced by #108476.

Rollup of 7 pull requests #108707 (Comparison Link)

(instructions:u)	mean	range	count
Regressions ❌ (primary)	0.7%	[0.6%, 0.7%]	2
Regressions ❌ (secondary)	1.5%	[1.3%, 1.7%]	6
Improvements ✅ (primary)	-0.4%	[-0.4%, -0.3%]	3
Improvements ✅ (secondary)	-0.7%	[-1.0%, -0.5%]	9
All ❌✅ (primary)	0.1%	[-0.4%, 0.7%]	5

• Regressions are just noise in the historically noisy benchmarks