Criterion 性能测试

← 上一篇 Benchmark

置信区间与离群值

实测 · 本机执行输出

  (含下载编译 criterion,第一次约 18 s)
  mix_1000                time:   [1.7172 µs 1.7185 µs 1.7199 µs]
  Found 10 outliers among 100 measurements (10.00%)
    2 (2.00%) low mild
    6 (6.00%) high mild
    2 (2.00%) high severe
  mix_scaling/100         time:   [138.35 ns 138.72 ns 139.15 ns]
                          thrpt:  [718.67 Melem/s 720.87 Melem/s 722.78 Melem/s]
  Found 12 outliers among 100 measurements (12.00%)
    5 (5.00%) high mild
    7 (7.00%) high severe
  mix_scaling/1000        time:   [1.7165 µs 1.7184 µs 1.7208 µs]
                          thrpt:  [581.14 Melem/s 581.94 Melem/s 582.58 Melem/s]
  Found 15 outliers among 100 measurements (15.00%)
    3 (3.00%) low mild
    8 (8.00%) high mild
    4 (4.00%) high severe
  mix_scaling/10000       time:   [17.553 µs 17.566 µs 17.579 µs]
                          thrpt:  [568.87 Melem/s 569.30 Melem/s 569.69 Melem/s]
  Found 6 outliers among 100 measurements (6.00%)
    2 (2.00%) low mild
    2 (2.00%) high mild
    2 (2.00%) high severe

#[bench] 与基准陷阱见Benchmark。criterion 0.7 从 crates.io 下载,这里关闭了默认 feature(没有 HTML 报告与绘图)。

误报

实测 · 本机执行输出

  mix_1000                time:   [1.7457 µs 1.7597 µs 1.7747 µs]
                          change: [+1.4886% +2.0427% +2.5868%] (p = 0.00 < 0.05)
                          Performance has regressed.
  mix_scaling/100         time:   [139.09 ns 139.84 ns 140.61 ns]
                   change:
                          time:   [+0.0356% +0.3578% +0.7148%] (p = 0.04 < 0.05)
                          Change within noise threshold.
  mix_scaling/1000        time:   [1.7353 µs 1.7455 µs 1.7564 µs]
                   change:
                          time:   [+0.6212% +0.9607% +1.3076%] (p = 0.00 < 0.05)
                          Change within noise threshold.
  mix_scaling/10000       time:   [18.086 µs 18.135 µs 18.179 µs]
                   change:
                          time:   [+2.8520% +3.1430% +3.4750%] (p = 0.00 < 0.05)
                          Performance has regressed.

这些数字是 macOS 上的实测,每次运行不同;不要把「regressed」当作一定是代码引起的。

真实的变化

实测 · 本机执行输出

  mix_1000                time:   [1.8086 µs 1.8148 µs 1.8212 µs]
                          change: [+2.7983% +3.4598% +4.1006%] (p = 0.00 < 0.05)
                          Performance has regressed.
  mix_scaling/100         time:   [143.61 ns 143.91 ns 144.22 ns]
                   change:
                          time:   [+3.1284% +3.5333% +3.9139%] (p = 0.00 < 0.05)
                          Performance has regressed.
  mix_scaling/1000        time:   [1.8073 µs 1.8129 µs 1.8185 µs]
                   change:
                          time:   [+4.4159% +4.8738% +5.3484%] (p = 0.00 < 0.05)
                          Performance has regressed.
  mix_scaling/10000       time:   [18.474 µs 18.526 µs 18.579 µs]
                   change:
                          time:   [+1.5724% +2.0446% +2.5044%] (p = 0.00 < 0.05)
                          Performance has regressed.

保存的数据

实测 · 本机执行输出

  找到 4 个 estimates.json: ['mix_1000/new/estimates.json', 'mix_scaling/100/new/estimates.json', 'mix_scaling/1000/new/estimates.json', 'mix_scaling/10000/new/estimates.json']
  mix_1000 的 mean: {'point_estimate': 1815.4}  置信区间(ns): [1809.6, 1821.4]  median: 1821.7  std_dev: 30.2

分步演示

Criterion

本步的对象

本文用到的工具与边界

本机 macOS 27 arm64 / rustc 1.98.1(brew)/ cargo 1.98.1;单个文件用 rustc 直接编译,需要依赖时用 cargo(经代理从 crates.io 下载);计时用 std::time::Instant,数字每次运行会有小幅抖动。criterion 0.7(default-features = false,仅 cargo_bench_support)
没有验证:HTML 报告与 gnuplot/plotters 图表、--save-baseline 与 --baseline 的对比流程、iter_batched 的各种批量模式、async 基准(需要 async 特性)、更长的 measurement-time 对误报的影响、criterion 的 noise_threshold 与 significance_level 的调整

下一篇 · RUST 161

Clippy 静态检查