Alerting & Regression Detection

A dashboard tells you something is wrong once you look at it. Alerting is what makes a performance regression a 40-minute incident instead of a fortnight of gradual decline nobody noticed. As set out in RUM Architecture, Tooling & Self-Hosting, the pipeline exists to produce trustworthy percentiles; this page covers what to do with them at three in the morning — how to model a baseline that tolerates daily traffic rhythm, which statistical test distinguishes a real shift from sampling noise, and how to route the result so that the person paged can act.

The failure mode to design against is not a missed regression. It is an alert channel everyone has muted. Every rule below is chosen for precision first: a page that fires only when something genuinely changed is worth more than five that catch everything.

What to alert on

Three families of signal, in descending order of how often they should page someone.

Volume collapse deserves special mention: a reporter that stops sending produces a beautiful, flat, entirely fictional metric. Alert on beacon count per route dropping more than 40% against the same hour last week, and you will catch a broken deploy before anyone notices the metric itself is a lie.

Threshold configuration

Thresholds are relative to a baseline, and the baseline has to carry the daily and weekly rhythm of your own traffic. A rolling median of the same hour on the previous 7 days is the cheapest model that works.

The suppression rule is as important as the firing rules. Any alert evaluated on a cohort with too few samples should not fire at all — see Calculating Confidence Intervals for a Sampled p75 for why a thousand is the rough floor.

Measurement implementation

The query shape matters more than the alerting tool. What follows computes the current window, the baseline and the ratio in one pass, so the alert condition is a single scalar comparison.

-- ClickHouse: current hour vs the same hour on the previous 7 days.
WITH
  now() - INTERVAL 1 HOUR AS window_start
SELECT
  route,
  quantileTDigestIf(0.75)(value, ts >= window_start)                    AS p75_now,
  countIf(ts >= window_start)                                           AS n_now,
  quantileTDigestIf(0.75)(value, ts <  window_start)                    AS p75_base,
  countIf(ts < window_start)                                            AS n_base,
  p75_now / nullIf(p75_base, 0)                                         AS ratio
FROM rum_events
WHERE metric = 'LCP'
  AND (
    ts >= window_start
    OR (toHour(ts) = toHour(now()) AND ts >= now() - INTERVAL 7 DAY)
  )
GROUP BY route
HAVING n_now >= 1000 AND ratio > 1.2
ORDER BY ratio DESC;

The HAVING clause is the whole design: no row is returned unless the window carries enough samples for the percentile to mean anything and the ratio exceeds the tolerance. An alert built on this query cannot fire on a quiet route at four in the morning, which is precisely when spurious pages do the most damage to a rota.

For a statistically defensible verdict rather than a ratio heuristic, a two-sample test on the underlying distributions is the right tool — the Mann-Whitney U test and the bootstrap approach are worked through in Detecting Vitals Regressions with Statistical Tests.

Step-by-step response workflow

1. Confirm the volume is intact. If beacon count fell alongside the metric moving, you are looking at an instrumentation failure, not a performance one. Fix the reporter first; the metric is not evidence until you do.
2. Check whether the shift is one cohort or all of them. A regression in every cohort simultaneously is nearly always infrastructure — origin, CDN or DNS. One cohort is nearly always code.
3. Diff the release marker. Group the alerting window by the release attached to each beacon. A clean split at a deploy boundary ends the investigation in two minutes.
4. Pull the sub-part composition for the affected route with the attribution build and compare it against the baseline window. The sub-part that moved names the subsystem.
5. Decide roll back or roll forward on the size of the regression and the size of the affected population, not on the metric name.
6. Close the loop. Every page that turns out to be noise should end with a rule change, and every regression that was not caught should end with a new rule.

Field-data analysis patterns

• Release marker as a first-class dimension. Without it, every regression investigation starts with archaeology in the deploy log. With it, the query answers the question directly.
• Alert on the cohort, report on the origin. Cohort-level rules fire earlier and more specifically; origin-level rules exist for the weekly report.
• Watch the rating mix, not only the percentile. A distribution can grow a fat Poor tail while the p75 barely moves — see the shift in the mix before the percentile catches up.
• Correlate against deploy and traffic events. Overlaying deploys, marketing sends and CDN configuration changes on the same timeline eliminates most of the guesswork.

Alert routing and ownership

An alert with no owner is a notification. Routing is part of the rule, not an afterthought.

The payload requirements are not decoration. A page that says “LCP regressed” costs the responder ten minutes of query writing before they can think; a page that says “LCP p75 on /products/[slug], mid-tier Android, 3.1 s vs 2.4 s baseline, 4,200 samples, releases 214 vs 213” starts the investigation at step four. The mechanics of getting that payload into a chat or paging tool are covered in Routing RUM Alerts to Slack and PagerDuty.

Choosing a baseline model

Everything above rests on a baseline, and the model you choose decides which regressions you can see at all. Four are worth knowing, in ascending order of cost.

Same hour, previous week. The default. It carries the daily and weekly rhythm of your traffic with no statistics at all, and it is robust to a single bad day because you take the median of the previous seven same-hour values. Its weakness is holidays and campaign days, which have no matching precedent.

Rolling median with a lag. A median of the previous N windows, offset so the current window is excluded. Cheaper to compute and adapts to a gradual change in the site. Its weakness is exactly that adaptation: a sustained regression is absorbed into the baseline within a day or two and the alert silently resolves while the site is still slow. Never use a short rolling baseline as the only model.

Seasonal decomposition. Fit a daily and weekly seasonal component and alert on the residual. This handles rhythm properly and flags anomalies that a same-hour comparison would miss, at the cost of a model that needs several weeks of history and periodic refitting.

Control chart on the ratio. Track the ratio of current to baseline over time and alert when it exceeds a multiple of its own standard deviation. This adapts the sensitivity to how noisy each route actually is, which stops high-traffic routes from being under-alerted and low-traffic ones from being over-alerted.

Whichever you pick, hold a second, much longer reference — the same window a year ago, or a frozen “known good” period — so that slow multi-month drift has something to be measured against. Every adaptive baseline is blind to the change that happens slowly enough.

What to put in the alert payload

The gap between a useful page and a useless one is almost entirely payload. An alert should answer, without a single follow-up query: what moved, for whom, by how much, since when, and what changed at that moment.

// alert-payload.mjs — assembled by the alert evaluator, consumed by any sink.
export function buildPayload(row, baselineRow, releases) {
  const delta = row.p75 - baselineRow.p75;
  return {
    title: `${row.metric} p75 regression on ${row.route}`,
    severity: row.ratio > 1.4 ? 'page' : 'ticket',
    facts: {
      route: row.route,
      cohort: `${row.device}/${row.network}`,
      current: `${Math.round(row.p75)} ms`,
      baseline: `${Math.round(baselineRow.p75)} ms`,
      change: `${delta > 0 ? '+' : ''}${Math.round(delta)} ms (${Math.round((row.ratio - 1) * 100)}%)`,
      samples: row.samples,
      windowStart: row.windowStart,
    },
    // The two releases seen in the alerting window, newest first. A clean
    // split here ends most investigations before they start.
    releases: releases.slice(0, 2),
    links: {
      dashboard: `${process.env.DASH}/route?route=${encodeURIComponent(row.route)}`,
      attribution: `${process.env.DASH}/attribution?route=${encodeURIComponent(row.route)}`,
    },
  };
}

Two fields earn their place repeatedly. samples lets the responder dismiss a low-volume artefact in two seconds. releases turns “something changed” into “release 214 changed”, which is the whole investigation on a good day.

Failure modes and gotchas

• Alerting on an average. A mean is dragged by outliers and hides the tail. Every rule here operates on a percentile or a share, never on a mean.
• A baseline that includes the regression. A rolling 24-hour baseline absorbs a sustained regression within a day and the alert silently resolves while the site is still slow. Use same-hour-last-week, and hold a longer reference window for weekly comparison.
• Ignoring traffic-mix shifts. A campaign that brings mobile traffic moves every percentile with no code change. Check cohort volumes before treating a shift as a regression.
• Sampling rate changes. Change the sample rate and every percentile moves for reasons that have nothing to do with the site. Version the sampling configuration and record it on the beacon.
• Timezone-naive baselines. Comparing 09:00 UTC against a baseline built in local time produces a phantom regression every daylight-saving transition.
• Alerting per URL instead of per route template. High-cardinality alerting produces thousands of rules, each below the sample floor, none of which is meaningful.

CI/CD integration

Field alerting catches what escaped; CI catches what should never ship. The two need the same budgets so a team is not told different things by different systems.

// scripts/check-budget.mjs — run after deploy, before marking a release healthy.
const BUDGETS = {
  '/': { LCP: 2500, INP: 200 },
  '/products/[slug]': { LCP: 2800, INP: 200 },
  '/checkout': { LCP: 3000, INP: 250 },
};

const res = await fetch(process.env.RUM_API + '/p75?window=2h&release=' + process.env.RELEASE);
const rows = await res.json();

let failed = false;
for (const row of rows) {
  const budget = BUDGETS[row.route]?.[row.metric];
  if (!budget) continue;
  if (row.samples < 500) {
    console.log(`skip ${row.route} ${row.metric}: ${row.samples} samples`);
    continue;
  }
  if (row.p75 > budget) {
    console.error(`FAIL ${row.route} ${row.metric} p75 ${row.p75}ms > ${budget}ms (n=${row.samples})`);
    failed = true;
  }
}
process.exit(failed ? 1 : 0);

Running this two hours after a deploy — against beacons carrying the new release marker only — turns a canary into a measurement rather than a hope. Budget definitions belong in one place, shared with the CI gate described in Performance Budgets & SLOs.

FAQ

How long should a regression persist before it pages someone?

Three consecutive windows at your evaluation granularity — around 45 minutes on 15-minute windows. Shorter than that and normal traffic variance will page you; longer and you are trading incident duration for quiet.

Can I alert on a custom metric the same way?

Yes, and the rules transfer directly — a named User Timing measure has the same baseline, persistence and sample-floor requirements as a standard vital. The only difference is that you own the definition, so a change to what the measure covers looks exactly like a regression. Version the metric name when its meaning changes.

Should alerts run on raw events or on rollups?

Rollups, always. Alert queries run continuously, and a rule that scans raw events every five minutes becomes the most expensive query in the warehouse. Compute quantile state hourly and alert against it.

Why does my alert fire every morning at the same time?

Because the rule compares against a global threshold or a short rolling baseline rather than the same hour last week. Traffic mix has a strong daily cycle — mobile share peaks at commute times — and the metric moves with it.

Is a percentage change enough, or do I need a statistical test?

A ratio rule with a sample floor is adequate for paging. A statistical test is what you want before telling a team their release caused a regression, because it quantifies how surprised you should be.

How do I stop deploys from generating a page every time?

Suppress evaluation for a short window after a deploy only if your releases are genuinely brief and frequent, and never suppress the volume check. A better answer is to attach the release marker to every beacon and evaluate per release: a regression that only affects the new release is exactly the alert you want, and it arrives with the cause already attached.

Should the same rules run against every cohort?

No. Run paging rules on the cohorts that carry enough volume to be stable — usually device tier crossed with the top handful of route templates. Everything else belongs in the daily budget check, where a wide comparison is cheap and nobody is woken up by a small sample.

What is a reasonable target for alert precision?

Aim for four out of five pages leading to a real finding. Below that, responders start closing alerts without reading them, and the next real regression is closed the same way. Track the disposition of every page for a month; it is the only way to know which rule is generating the noise.

What do I do about routes with too little traffic to alert on?

Aggregate them into a template group and alert on the group, or accept a daily budget check instead of real-time alerting. Do not lower the sample floor — the alert will fire on noise and be muted within a week.

Good alerting is mostly restraint. Every rule added should displace one that no longer earns its place, and the measure of the system is not how many regressions it can theoretically catch but how many pages a responder still reads carefully after a year on the rota.

Related

• Grafana Dashboards for Web Performance — the panels these rules are evaluated against.
• Detecting Vitals Regressions with Statistical Tests — bootstrap and rank tests for a defensible verdict.
• Routing RUM Alerts to Slack and PagerDuty — getting an actionable payload to the responder.
• Performance Budgets & SLOs — the targets these alerts defend.
• RUM Data Sampling Strategies — why the sample floor exists and how to keep cohorts above it.