Detecting Vitals Regressions with Statistical Tests

A ratio rule — “alert if p75 is 20% above baseline” — is the right tool for paging someone. It is the wrong tool for telling a team their release caused a regression, because it cannot distinguish a real shift from the variation a metric shows anyway. This guide sits under Alerting & Regression Detection and covers three tests that work on RUM distributions, when each applies, and how to express the result so that it survives the meeting.

Prerequisites

Raw or sampled event data for two windows: the candidate period and a baseline. Pre-aggregated percentiles cannot be tested.
A release marker on every beacon, so the two windows are actually comparable populations.
The sample-size floors from Calculating Confidence Intervals for a Sampled p75, because none of these tests rescues a cohort with two hundred observations.

How to test it

Step 1 — Choose the test that matches the question

Test	Answers	Assumes	Cost
Mann-Whitney U	Are the two distributions shifted?	Independence; ordinal data	Cheap
Bootstrap on the p75 difference	How big is the shift, with an interval?	Independence, or cluster-aware resampling	Moderate
Kolmogorov-Smirnov	Did the distribution shape change?	Independence	Cheap
Sequential (alpha-spending)	Can I stop early?	Pre-declared plan	Moderate

A t-test is missing from that list deliberately. Web-vitals distributions are heavily right-skewed, and a test built around means is dominated by the tail rather than by the shift you care about.

Step 2 — Run the rank test for a yes-or-no answer

The Mann-Whitney U test asks whether a randomly drawn value from one sample tends to be larger than one from the other. It makes no distributional assumptions, which suits a metric with a heavy right tail.

// Mann-Whitney U with a normal approximation — adequate above ~200 per group.
export function mannWhitney(a, b) {
  const all = [...a.map((v) => ({ v, g: 0 })), ...b.map((v) => ({ v, g: 1 }))]
    .sort((x, y) => x.v - y.v);

  // Average ranks for ties, which are common in millisecond-rounded data.
  let i = 0;
  while (i < all.length) {
    let j = i;
    while (j + 1 < all.length && all[j + 1].v === all[i].v) j++;
    const rank = (i + j + 2) / 2;
    for (let k = i; k <= j; k++) all[k].rank = rank;
    i = j + 1;
  }

  const n1 = a.length, n2 = b.length;
  const r1 = all.filter((x) => x.g === 0).reduce((s, x) => s + x.rank, 0);
  const u1 = r1 - (n1 * (n1 + 1)) / 2;
  const mean = (n1 * n2) / 2;
  const sd = Math.sqrt((n1 * n2 * (n1 + n2 + 1)) / 12);
  const z = (u1 - mean) / sd;
  // Two-sided p-value from the standard normal.
  const p = 2 * (1 - normalCdf(Math.abs(z)));
  return { u: u1, z, p, effect: u1 / (n1 * n2) };   // effect: P(a > b)
}

function normalCdf(x) {
  // Abramowitz-Stegun approximation; accurate to ~7 decimal places.
  const t = 1 / (1 + 0.2316419 * x);
  const d = 0.3989423 * Math.exp((-x * x) / 2);
  const p = d * t * (0.3193815 + t * (-0.3565638 + t * (1.781478 + t * (-1.821256 + t * 1.330274))));
  return 1 - p;
}

The effect value matters more than the p-value. With a hundred thousand beacons per window, a 3 ms shift is statistically significant and completely irrelevant. An effect near 0.5 means the distributions overlap almost entirely regardless of what the p-value says.

Step 3 — Bootstrap the p75 difference for a number people can act on

Significance says something changed. The bootstrap says how much, with an interval, in the units the team already uses.

// Cluster bootstrap on the p75 difference: resample sessions, not events.
export function bootstrapP75Diff(baselineSessions, candidateSessions, iterations = 2000) {
  const p75 = (xs) => {
    const s = xs.slice().sort((x, y) => x - y);
    return s[Math.floor(s.length * 0.75)];
  };
  const drawFlat = (sessions) => {
    const out = [];
    for (let i = 0; i < sessions.length; i++) {
      const s = sessions[Math.floor(Math.random() * sessions.length)];
      for (const v of s.values) out.push(v);
    }
    return out;
  };

  const diffs = [];
  for (let i = 0; i < iterations; i++) {
    diffs.push(p75(drawFlat(candidateSessions)) - p75(drawFlat(baselineSessions)));
  }
  diffs.sort((x, y) => x - y);
  return {
    estimate: diffs[Math.floor(iterations * 0.5)],
    lower: diffs[Math.floor(iterations * 0.025)],
    upper: diffs[Math.floor(iterations * 0.975)],
  };
}

The output — “release 214 raised p75 LCP by 310 ms (95% interval 240 to 380)” — is a sentence that ends a debate. A p-value alone starts one.

Three of four methods flag it. Only the bootstrap says how large the change is and how confident you should be — which is the part a team can act on.

Step 4 — Check the shape, not only the shift

A distribution can grow a heavy tail while the p75 barely moves — a minority of sessions getting much worse. A Kolmogorov-Smirnov test on the two empirical distributions catches that, and comparing rating shares often makes it visible without any test at all.

// KS statistic: the largest gap between two empirical distributions.
export function ksStatistic(a, b) {
  const xs = [...new Set([...a, ...b])].sort((x, y) => x - y);
  const cdf = (sample, v) => sample.filter((x) => x <= v).length / sample.length;
  let d = 0;
  for (const v of xs) d = Math.max(d, Math.abs(cdf(a, v) - cdf(b, v)));
  const n = (a.length * b.length) / (a.length + b.length);
  return { d, approxP: 2 * Math.exp(-2 * n * d * d) };
}

The p75 moved 4%. The Poor share more than doubled — a change concentrated in a minority of sessions, which is exactly what a shape test detects and a percentile does not.

Step 5 — Do not peek without paying for it

Checking a test repeatedly as data arrives inflates the false-positive rate: at 95% confidence, twenty looks produce roughly one spurious significant result by construction. Either declare a fixed window and test once, or use a sequential design that spends the error budget deliberately.

// A simple alpha-spending schedule: tighten the threshold for early looks.
export function sequentialThreshold(look, totalLooks, alpha = 0.05) {
  // O'Brien-Fleming style: very strict early, approaching alpha at the end.
  const fraction = look / totalLooks;
  return alpha * Math.pow(fraction, 3);
}

In practice, most teams are better served by the discipline than the mathematics: decide the window before the release, test once at the end of it, and treat anything you notice earlier as a reason to look rather than a result.

Step 6 — Report it in a form that travels

The output of any of these tests should be a single sentence with four parts: the population, the direction, the size, and the uncertainty. “On mid-tier mobile sessions on /products/[slug] (n=41,200), release 214 raised p75 LCP by 310 ms, 95% interval 240 to 380.” Anything shorter invites the wrong question; anything longer will be summarised by someone else, badly.

When not to reach for a test

Three situations where the statistics are not the problem.

The volume collapsed. If beacon count fell alongside the metric, you are comparing two different populations and no test on that comparison means anything. Fix the instrumentation first.

The traffic mix changed. A campaign that brings mobile traffic shifts every percentile with no code change. Stratify by cohort before testing, or you will detect the marketing calendar.

The effect is obvious. A p75 that doubled does not need a test. Spending an afternoon on a bootstrap to confirm what a chart already shows is a way of not fixing the problem.

The tests earn their keep in the middle band — changes of 5–15% on cohorts of moderate size, where the eye cannot tell and the decision matters.

Verifying it works

A null test on two halves of the same baseline returns a non-significant result. Run this first; if random halves of your baseline test as different, the observations are not independent.
A synthetic 10% shift is detected reliably at your typical sample sizes.
Cluster resampling widens the interval compared with naive resampling. If it does not, sessions really are contributing one value each.
Effect size is reported alongside every p-value, in the metric’s own units.
The window is declared before the test, and the number of looks is recorded.

Sample size decides which questions the data can answer. Below a thousand per window, only large regressions are detectable at all — and those rarely need a test.

Edge cases and gotchas

Significance without relevance. At large n, everything is significant. Always pair with an effect size and a materiality threshold agreed in advance.
Sessions are clusters. A user generating twenty page views contributes correlated values; naive resampling understates uncertainty.
Mixed sampling rates between windows invalidate the comparison entirely. Version the sampling configuration and filter on it.
Bots and synthetic monitoring show up as an unusually tight cluster of fast values and can mask a real regression. Filter before testing.
Rounded values create heavy ties. Millisecond rounding is fine; the rank test needs the tie correction shown above, which is easy to omit.
Multiple cohorts, multiple tests. Testing twenty cohorts means expecting a false positive. Adjust the threshold or state that you did not.

Alerting & Regression Detection — the cheap rules that page someone, which these tests follow up.
Calculating Confidence Intervals for a Sampled p75 — the sample sizes every test here depends on.
User Impact Mapping — the same statistical care applied to business outcomes.