Routing RUM Alerts to Slack and PagerDuty

An alert that arrives without context costs the responder ten minutes of query writing before they can think. An alert that arrives too often costs you the channel entirely. This guide sits under Alerting & Regression Detection and covers the payload a performance alert needs, how to route by severity without duplicating, deduplication and auto-resolution, and how to measure whether the alerts you send are worth sending.

Prerequisites

  • Evaluated alert conditions with the sample floors and persistence rules from the parent guide — this is the delivery layer, not the detection layer.
  • A Slack incoming webhook or app token, and a PagerDuty Events API v2 routing key.
  • A dashboard URL scheme that accepts route and cohort parameters, so links can be constructed rather than hand-made.

How to build the delivery layer

Step 1 — Build one payload, render it per destination

The evaluator should produce a single structured alert object. Formatting for chat or paging is a rendering concern, and keeping them separate is what stops the two channels from drifting.

// alert.js — the canonical shape every destination renders from.
export function buildAlert(row, baseline, releases) {
  const deltaMs = Math.round(row.p75 - baseline.p75);
  const pct = Math.round((row.p75 / baseline.p75 - 1) * 100);
  return {
    // A stable key: the same condition must produce the same key every time,
    // or deduplication and auto-resolve cannot work.
    key: `rum:${row.metric}:${row.route}:${row.deviceTier}`,
    severity: pct >= 40 ? 'critical' : pct >= 20 ? 'warning' : 'info',
    title: `${row.metric} p75 up ${pct}% on ${row.route}`,
    facts: {
      route: row.route,
      cohort: `${row.deviceTier} / ${row.network}`,
      current: `${Math.round(row.p75)} ms`,
      baseline: `${Math.round(baseline.p75)} ms`,
      change: `${deltaMs > 0 ? '+' : ''}${deltaMs} ms (${pct}%)`,
      samples: row.samples,
      window: row.windowLabel,
      releases: releases.slice(0, 2).join(' → '),
    },
    links: {
      dashboard: `${process.env.DASH}/route?route=${encodeURIComponent(row.route)}&tier=${row.deviceTier}`,
      attribution: `${process.env.DASH}/attribution?route=${encodeURIComponent(row.route)}`,
      runbook: 'https://wiki.example.com/runbooks/rum-regression',
    },
  };
}

Two fields carry disproportionate weight. samples lets a responder dismiss a low-volume artefact in two seconds. releases turns “something changed” into “release 214 changed” on a good day, which is most days.

Step 2 — Route by severity, to one destination each

Sending everything everywhere is how a channel becomes noise. One destination per severity, with a documented meaning.

Severity Destination Expectation
critical PagerDuty, immediate Someone is woken; a human responds now
warning Slack channel, working hours Triaged within a day
info Daily digest Read in the weekly review 
// route.js — one destination per severity. No fan-out.
export async function dispatch(alert) {
  if (alert.severity === 'critical') return sendPagerDuty(alert);
  if (alert.severity === 'warning') return sendSlack(alert, process.env.SLACK_PERF);
  return queueForDigest(alert);
}

Step 3 — Render for Slack so it reads at a glance

export async function sendSlack(alert, webhook) {
  const f = alert.facts;
  const body = {
    text: alert.title,                       // fallback for notifications
    blocks: [
      { type: 'header', text: { type: 'plain_text', text: alert.title } },
      {
        type: 'section',
        fields: [
          { type: 'mrkdwn', text: `*Cohort*\n${f.cohort}` },
          { type: 'mrkdwn', text: `*Change*\n${f.change}` },
          { type: 'mrkdwn', text: `*Now / baseline*\n${f.current} / ${f.baseline}` },
          { type: 'mrkdwn', text: `*Samples*\n${f.samples.toLocaleString('en-US')}` },
          { type: 'mrkdwn', text: `*Window*\n${f.window}` },
          { type: 'mrkdwn', text: `*Releases*\n${f.releases || 'unknown'}` },
        ],
      },
      {
        type: 'actions',
        elements: [
          { type: 'button', text: { type: 'plain_text', text: 'Dashboard' }, url: alert.links.dashboard },
          { type: 'button', text: { type: 'plain_text', text: 'Attribution' }, url: alert.links.attribution },
          { type: 'button', text: { type: 'plain_text', text: 'Runbook' }, url: alert.links.runbook },
        ],
      },
    ],
  };
  await fetch(webhook, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(body),
  });
}

Step 4 — Use PagerDuty deduplication properly

The Events API v2 deduplicates on dedup_key. Sending the same key while an incident is open updates it instead of creating a second one, and sending a resolve event with that key closes it. That combination is what makes an alert self-clearing.

export async function sendPagerDuty(alert, action = 'trigger') {
  await fetch('https://events.pagerduty.com/v2/enqueue', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      routing_key: process.env.PD_ROUTING_KEY,
      event_action: action,                    // trigger | acknowledge | resolve
      dedup_key: alert.key,                    // stable per condition
      payload: {
        summary: alert.title,
        severity: 'error',
        source: 'rum-alerting',
        component: alert.facts.route,
        group: alert.facts.cohort,
        custom_details: alert.facts,
      },
      links: Object.entries(alert.links).map(([text, href]) => ({ text, href })),
    }),
  });
}

// When the condition clears, resolve rather than leaving it to a human.
export const resolveAlert = (alert) => sendPagerDuty(alert, 'resolve');

Step 5 — Auto-resolve, and require the same persistence to clear

An alert that clears on the first good window will flap. Require the condition to be healthy for as many consecutive windows as it took to fire.

// state.js — fire after N bad windows, resolve after N good ones.
const REQUIRED = 3;
const state = new Map();

export function evaluate(key, isBad) {
  const s = state.get(key) ?? { bad: 0, good: 0, open: false };
  if (isBad) { s.bad++; s.good = 0; } else { s.good++; s.bad = 0; }

  let action = null;
  if (!s.open && s.bad >= REQUIRED) { s.open = true; action = 'trigger'; }
  if (s.open && s.good >= REQUIRED) { s.open = false; action = 'resolve'; }

  state.set(key, s);
  return action;
}
Alert volume per month by delivery designDetection did not change at all between these four. The entire reduction is delivery design — and the last configuration is the only one people still read.Alert volume per month by delivery designSame detection rules, four delivery configurationsFan-out to every channel214Severity routing only86+ deduplication31+ auto-resolve and persistence9062.5125187.5250
Detection did not change at all between these four. The entire reduction is delivery design — and the last configuration is the only one people still read.

Step 6 — Measure precision and prune

An alerting system without feedback degrades. Record the disposition of every alert and review it monthly.

// A one-field feedback loop: what happened to this alert?
export async function recordDisposition(key, disposition, note) {
  // disposition: 'real' | 'noise' | 'duplicate' | 'expected'
  await fetch(process.env.WAREHOUSE_URL + '/alert_dispositions', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ key, disposition, note, at: new Date().toISOString() }),
  });
}

The target is four real findings in every five alerts. Below that, responders start closing alerts without reading them, and the next genuine regression is closed the same way. Any rule with a majority of noise dispositions should be tightened or deleted at the monthly review — deleting an alert rule is a legitimate outcome and is nearly always under-used.

What a good alert looks like in practice

The difference between a useful page and a useless one is visible in a single line of text.

Useless: “LCP regression detected.”

Useful: “LCP p75 up 34% on /products/[slug] — mid-tier mobile, 3.1 s vs 2.3 s baseline, 4,200 samples in the last hour, releases 214 → 213.”

The second version answers the responder’s first four questions before they ask them: what, for whom, how much, and what changed. It takes no more engineering than the first — the facts were already in the evaluator’s result set. The only decision was to put them in the payload.

The same principle applies to the links. A dashboard link that lands on a filtered view of the affected route and cohort saves more responder time than any other single field, because the alternative is reconstructing that filter by hand at three in the morning.

What happened to the alerts that firedThe disposition log is the only honest measure of an alerting system. Before tuning, four in ten pages were noise — which is how a channel gets muted.What happened to the alerts that firedDisposition of every alert over three months, before and after tuningBefore tuning19%8%31%42%After tuning74%12%6%8%Real findingExpected (known work)DuplicateNoise
The disposition log is the only honest measure of an alerting system. Before tuning, four in ten pages were noise — which is how a channel gets muted.

Verifying it works

  1. A test alert reaches both destinations with every field populated, and the links open on the correct filtered view.
  2. Repeated evaluations of the same condition update one incident, rather than creating several.
  3. Conditions that clear resolve automatically within the persistence window.
  4. Precision above 80%, measured from the disposition data over a month.
  5. No alert fires below the sample floor. Query the disposition table for alerts with low sample counts; there should be none.
One evaluation, one destinationEvery stage exists to prevent a specific failure: firing on noise, firing on a spike, arriving without context, and arriving in four places at once.One evaluation, one destinationSeverity decides the route; nothing fans outCondition evaluatedwith sample floorPersistence check3 windows to firePayload builtfacts, releases, linksSingle destinationpage, channel or digest
Every stage exists to prevent a specific failure: firing on noise, firing on a spike, arriving without context, and arriving in four places at once.

Edge cases and gotchas

  • Unstable dedup keys. Including a timestamp or a value in the key creates a new incident every evaluation. The key must describe the condition, not the observation.
  • Alert storms at deploy time. A release that legitimately changes many routes fires many alerts. Group by release, or suppress non-critical alerts briefly after a deploy — never suppress the volume check.
  • Webhook rate limits. Slack and PagerDuty both throttle. Batch info-level alerts into a digest rather than sending each one.
  • Secrets in payloads. Alert facts are pasted into chat and tickets. Never include raw URLs with query strings, which may contain tokens.
  • Timezones in the window label. A responder in another region reading a window labelled in server-local time will investigate the wrong hour. Use ISO timestamps with an explicit zone.
  • Nobody owns the channel. An alert channel without a named owner accumulates rules nobody reviews. Assign it, and review monthly.

FAQ

Who should own the alert channel?

The team that owns the routes being alerted on, with one named person responsible for the monthly review. A channel owned by a central platform team collects rules nobody prunes, because nobody is woken by them.

Should performance regressions page someone at night?

Only the severe ones, and only where a human can act. A 40% p75 regression affecting checkout deserves a page; a 12% drift on a marketing route does not.

How do I stop alerts firing after every deploy?

Attach the release marker to every beacon and evaluate per release, so a regression is reported with its cause attached. Blanket post-deploy suppression hides exactly the alerts you most want.

What belongs in the payload?

Route, cohort, current and baseline values, the change, the sample count, the window, the release diff, and links to a pre-filtered dashboard and the runbook. Anything else is decoration.

Is Slack enough, or do I need paging?

Chat is enough for anything that can wait until morning, which is most performance regressions. Paging is for the ones where every hour has a measurable cost, and those should be rare enough that the page is taken seriously.