rules/inference.md
GitHubInference: comparing two histograms
All methods take another Histogram (a RecordableHistogram, an
ELDHistogram, or a snapshot). Results are null-prototype objects. Signs always
mean this minus other.
| Method | Versions | Returns |
|---|---|---|
welchTest(other[, { confidence = 0.95 }]) |
v26.8.0, v24.21.0 | { tStatistic, degreesOfFreedom, pValue, confidenceInterval: { lower, upper } } |
mannWhitneyTest(other) |
v26.8.0, v24.21.0 | { uStatistic, zScore, pValue } |
ksTest(other) |
v26.8.0, v24.21.0 | number D in [0, 1]; 0 means identical |
cohensD(other) |
v26.8.0, v24.21.0 | number; positive means this mean is higher |
cliffsD(other) |
v26.8.0, v24.21.0 | number in [-1, 1]; P(this > other) - P(other > this) |
meanCI([{ confidence = 0.95 }]) |
v26.9.0, v24.22.0 | { mean, lower, upper } (Student's t) |
percentileCI(p[, { confidence = 0.95 }]) |
v26.8.0, v24.21.0 | { value, lower, upper } (exact binomial) |
Verified behavior (main and v26.13.0):
welchTest:pValueis two-tailed.confidenceIntervalbounds the difference of means (this.mean - other.mean). Ifupper < 0, this is significantly lower. Iflower > 0, it is significantly higher.degreesOfFreedomis Welch-Satterthwaite.mannWhitneyTest:uStatisticcounts pairs where this > other (ties count half).zScore < 0means this tends to be lower. It uses the normal approximation with tie correction.- With fewer than 2 values in either histogram,
welchTestreturns{ tStatistic: 0, degreesOfFreedom: 0, pValue: 1, confidenceInterval: { lower: 0, upper: 0 } }andcohensDreturns0. - An empty
meanCI()returns{ mean: NaN, lower: NaN, upper: NaN }. With one value,lowerandupperareNaN. When all values are equal, both equalmean. - With fewer than 2 values,
percentileCIreturnslower === upper === value, and an empty histogram gives all0.valueequalspercentile(p). confidencemust be in (0, 1).
Choosing
- Latency is usually right-skewed and multimodal. Use
mannWhitneyTestorksTestfirst. UsewelchTestfor means of many roughly normal samples, such as per-run throughput. - Always report an effect size (
cohensDorcliffsD) next to a p-value. With thousands of samples, tiny differences become "significant". - When comparing many benchmarks, correct for multiple tests (for example,
Bonferroni:
pValue < 0.05 / n). - HDR buckets quantize values to
figuressignificant digits (default 3), and the stats run on the bucket values, not the raw values.
Feeding benchmark results
Benchmark rates are floats, so record an integer unit, for example ns per
operation: h.record(Math.round(Number(s.duration_ns) / s.operations)).
See the node-bench skill for where the samples come from.