2021-05-18 Triage Log

A lot of noise in the benchmark results this week. We are discussing (zulip archive, live zulip) how best to update the benchmark set to eliminate the noisy cases that are bouncing around. Beyond that, some large improvements to a few individual benchmarks.

The memory usage (max-rss) seemed largely flat. Except for an upward trend on tuple-stess that indicates 4% more memory from a week ago.

Triage done by @pnkfelix. Revision range: 382f748f23979e37e3e012b090e5a0313463f182..25a277f03df7e44643ddfcc240d034409cb2f505

5 Regressions, 7 Improvements, 2 Mixed 1 of them in rollups

Regressions

Reachable statics have reachable initializers #84549

  • Moderate regression in instruction counts (up to 1.4% on incr-unchanged builds of ctfe-stress-4-opt)
  • ctfe-stress-4-debug also saw a 1.2% regression in its incr-unchanged builds.
  • Other benchmarks were not significantly impacted.
  • The two regressions were anticipated from a perf run on the PR itself.

Improvements

have on_completion record subcycles #85186

  • Large improvement in instruction counts (up to -9.4% on incr-unchanged builds of syn-opt)
  • No other benchmark was significantly impacted.
  • The improvement was anticipated from a perf run on the PR itself.
  • (But this PR was about incr-comp correctness, not performance.)

Store VariantIdx to distinguish enum variants #85195

  • Very large improvement in instruction counts (up to -24.4% on full builds of match-stress-enum-check)
  • All of the match-stress-enum variants observed a similar improvement. This makes sense since that is a microbenchmark stressing the compilation of match expressions.
  • The improvement was anticipated from a perf run on the PR itself.

Do not allocate or unwind after fork #81858

  • Moderate improvement in instruction counts (up to -1.1% on incr-patched: b9b3e592dd cherry picked builds of style-servo-debug)
  • No other benchmark was significantly impacted.
  • Seems like noise.

Mixed

BTree: no longer copy keys and values before dropping them #84904

  • Moderate improvement in instruction counts (up to -1.2% on incr-unchanged builds of ctfe-stress-4-check and ctfe-stress-4-opt and ctfe-stress-4-debug)
  • Moderate regression in instruction counts (up to 1.2% on incr-patched: println builds of cargo-opt)
  • These changes were anticipated from a perf run on the PR itself.

rustc_codegen_ssa: only create backend BasicBlocks as-needed. #84993

  • Moderate improvement in instruction counts (up to -4.3% on full builds of syn-opt)
  • Moderate regression in instruction counts (up to 1.8% on full builds of deeply-nested-async-check)
  • The improvement to syn-opt was anticipated from perf runs on the PR itself.
  • The regression on deeply-nested-async-check was not predicted by that run.

Suspicious Noise

coercions

rustc_driver cleanup #83610

  • Moderate regression in instruction counts (up to 1.7% on full builds of coercions-debug)
  • No other benchmark was significantly impacted.
  • No performance regression was expected here.
  • Seems like noise.

Add auto traits and clone trait migrations for RFC2229 #84730

  • Moderate regression in instruction counts (up to 1.4% on full builds of coercions-debug)
  • No other benchmark was significantly impacted.
  • No performance regression was expected here.
  • Seems like noise.

Provide ExitStatusError #82973

  • Moderate regression in instruction counts (up to 1.7% on full builds of coercions-debug)
  • No other benchmark was significantly impacted.
  • No performance regression was expected here.
  • Seems like noise.

Fix --remap-path-prefix not correctly remapping rust-src component paths and unify handling of path mapping with virtualized paths #83813

  • Moderate improvement in instruction counts (up to -1.4% on full builds of coercions-debug)
  • No other benchmark was significantly impacted.
  • Seems like noise.

Rollup of 5 pull requests #85231

  • Moderate improvement in instruction counts (up to -1.7% on full builds of coercions-debug)
  • No other benchmark was significantly impacted.
  • Seems like noise.

Rollup of 8 pull requests #85414

  • Moderate improvement in instruction counts (up to -1.1% on incr-full builds of coercions-debug)
  • coercions-check and coercions-opt had smaller improvements, -0.7% and -0.5%, respectively.
  • No other benchmark was significantly impacted.
  • Seems like noise.
html5ever

Add support for const operands and options to global_asm! #84107

  • Moderate regression in instruction counts (up to 1.4% on full builds of html5ever-opt)
  • No other benchmark was significantly impacted.
  • No performance regression was expected here.
  • Seems like noise.

Update cc crate #85190

  • Moderate improvement in instruction counts (up to -1.2% on full builds of html5ever-opt)
  • coercions-debug also saw a -1.0% improvement on its full builds.
  • Based on the corresponding regressions above, seems like noise.

Nags requiring follow up

  • Double-check whether regression to deeply-nested-async-check from #84993 is noise or something real.

  • Look into the 4% tuple-stress regression. The first seems like gradual creep, plus a 3% jump that seems to be associated with #84571, “Parse unnamed fields of struct and union type.” (Perhaps its inherent given the nature of that microbenchmark, but some other benchmarks also had their memory usage impacted.