Gating Pull Requests on RUM Budgets

A performance gate is worth having only if it fails when something got slower and passes when it did not. Most gates fail at the second half: they flake, people learn to re-run them, and within a month the check is decorative. This guide sits under Performance Budgets & SLOs and covers a two-stage gate — a deterministic lab check on every pull request and a field verification after deploy — plus the variance handling and escape hatch that keep it trusted.

Prerequisites

  • Per-route budgets with a measured lab-to-field ratio, from Setting Per-Route Performance Budgets.
  • A preview deployment per pull request, or a way to build and serve the branch.
  • A release marker on every RUM beacon, so post-deploy verification can filter to the new build.

How to build the gate

Step 1 — Gate on resources first, because they are deterministic

Bundle size and request count do not vary between runs. They catch the most common regression — someone imported a library — with zero flakiness, and they run in seconds.

// scripts/check-resources.mjs — runs on every pull request, no browser needed.
import { statSync, readdirSync } from 'node:fs';
import { budgets } from '../budgets.config.js';

const bytes = (dir) => readdirSync(dir, { recursive: true })
  .filter((f) => f.endsWith('.js'))
  .reduce((sum, f) => sum + statSync(`${dir}/${f}`).size, 0);

let failed = false;
for (const [route, cfg] of Object.entries(budgets)) {
  const actual = bytes(`dist/routes/${cfg.chunkDir}`);
  if (actual > cfg.lab.jsBytes) {
    console.error(`FAIL ${route}: ${(actual / 1024).toFixed(1)} KB > ${(cfg.lab.jsBytes / 1024).toFixed(1)} KB`);
    failed = true;
  } else {
    console.log(`ok   ${route}: ${(actual / 1024).toFixed(1)} KB`);
  }
}
process.exit(failed ? 1 : 0);

Report the delta against the base branch rather than only the absolute value. “+38 KB on the product route” is a review comment; “190 KB” is a number nobody can judge.

Step 2 — Run the lab metrics with enough repetitions

Metrics from a headless browser vary between runs by 10–20% on shared CI infrastructure. A gate built on a single run flakes, and a flaky gate is worse than no gate.

// lighthouserc.js — median of five runs, budgets imported from one place.
import { budgets } from './budgets.config.js';

export default {
  ci: {
    collect: {
      numberOfRuns: 5,                          // median of five, never one
      url: Object.values(budgets).map((b) => `${process.env.PREVIEW_URL}${b.samplePath}`),
      settings: {
        // Match the cohort the budget was derived from, not the CI runner.
        throttling: { cpuSlowdownMultiplier: 4, rttMs: 150, throughputKbps: 1638 },
      },
    },
    assert: {
      assertions: {
        'largest-contentful-paint': ['error', { maxNumericValue: 1800, aggregationMethod: 'median' }],
        'total-blocking-time': ['error', { maxNumericValue: 300, aggregationMethod: 'median' }],
        'cumulative-layout-shift': ['error', { maxNumericValue: 0.05, aggregationMethod: 'median' }],
        'categories:performance': 'off',        // never gate on the composite score
      },
    },
  },
};

Gating on the composite score is the mistake to avoid: it is a weighted blend whose weights change between tool versions, so a version bump can fail a build that changed nothing.

Flake rate by gate designAbove about 5% flake, people start re-running the check reflexively, and a real failure is re-run with it.Flake rate by gate designShare of pull requests failing with no real regression, over three months1 run, shared runner22.4%5 runs, shared runner8.1%5 runs, dedicated runner2.6%Resource budget only0%0%6.25%12.5%18.75%25%
Above about 5% flake, people start re-running the check reflexively, and a real failure is re-run with it.

Step 3 — Comment on the pull request rather than only failing

A red check with no explanation sends the author to a log. A comment with the numbers sends them to the code.

// scripts/comment.mjs — one table, base against head.
export function renderComment(results) {
  const rows = results.map((r) => {
    const delta = r.head - r.base;
    const sign = delta > 0 ? '+' : '';
    const flag = r.head > r.budget ? ' ⚠️' : '';
    return `| ${r.route} | ${r.metric} | ${r.base} | ${r.head} | ${sign}${delta}${flag} | ${r.budget} |`;
  });
  return [
    '### Performance check',
    '',
    '| Route | Metric | Base | This PR | Delta | Budget |',
    '|---|---|---|---|---|---|',
    ...rows,
    '',
    '_Median of 5 runs at 4× CPU throttling. Budgets derived from field data; see budgets.config.js._',
  ].join('\n');
}

Step 4 — Verify in the field after deploy

The lab gate is a proxy. Two hours after a release, check the beacons carrying the new release marker against the field target — the only measurement that reflects real users.

// scripts/verify-release.mjs — non-blocking, informational, fast to read.
import { budgets } from '../budgets.config.js';

const res = await fetch(`${process.env.RUM_API}/p75?window=2h&release=${process.env.RELEASE}`);
const rows = await res.json();

const report = [];
for (const row of rows) {
  const cfg = budgets[row.route];
  if (!cfg || row.metric !== cfg.field.metric) continue;
  if (row.samples < 500) { report.push(`skip ${row.route}: ${row.samples} samples`); continue; }
  const verdict = row.p75 > cfg.field.target ? 'OVER' : 'ok';
  report.push(`${verdict} ${row.route} ${row.metric} ${Math.round(row.p75)}ms vs ${cfg.field.target}ms (n=${row.samples})`);
}
console.log(report.join('\n'));

Do not block a release on this. It lags, it is subject to traffic mix, and a rollback triggered by two hours of thin data is more likely to be wrong than right. Post it to the release channel and let the burn-rate alerting in Alerting & Regression Detection handle sustained problems.

Step 5 — Provide an escape hatch, and log it

Sometimes a regression is a deliberate trade. A gate with no override is bypassed by disabling it, which is the outcome you least want.

# .github/workflows/perf.yml — an override that leaves a trail.
- name: Performance budget
  id: perf
  run: node scripts/check-resources.mjs && npx lhci autorun
  continue-on-error: ${{ contains(github.event.pull_request.labels.*.name, 'perf-budget-override') }}

- name: Record override
  if: steps.perf.outcome == 'failure'
  run: |
    node scripts/record-override.mjs \
      --pr "${{ github.event.pull_request.number }}" \
      --author "${{ github.actor }}" \
      --reason "${{ github.event.pull_request.title }}"

The label makes the override deliberate and visible in review; the log makes it countable. Reviewing overrides monthly is what tells you whether a budget is wrong — three overrides on the same route in a quarter is a budget problem, not a discipline problem.

The two-stage gateOnly the first two block. The field check exists to catch what the lab could not see, and to keep the lab budgets honest over time.The two-stage gateDeterministic first, authoritative secondResource checkseconds, blockingLab metricsmedian of 5, blockingDeployrelease marker attachedField check2h later, informational
Only the first two block. The field check exists to catch what the lab could not see, and to keep the lab budgets honest over time.

Keeping the gate trusted

A performance gate lives or dies on whether engineers believe it. Four practices decide that.

Keep the flake rate under about 5%. Above that, re-running becomes reflexive and real failures are re-run away. A dedicated runner is the single most effective purchase here.

Fail with a number and a delta. “LCP 2,140 ms, budget 1,800 ms, +310 ms versus base” tells the author what to look for. “Performance check failed” tells them to ask someone.

Make the budget file reviewable. When a budget changes, it changes in a pull request with a reason, seen by the same people the gate applies to.

Review overrides, not just failures. The override log is the most honest feedback the system produces: it shows exactly where the budgets and the work are in tension.

The failure mode to watch for is a gate that has never failed. That is not a well-performing codebase; it is a budget set above where the code already sits, and it will not catch the regression it was built for.

Verifying it works

  1. A deliberately heavy pull request fails, and a no-op pull request passes ten times in a row. Test both before trusting it.
  2. Flake rate under 5%, measured from re-run data over a month.
  3. The comment appears on every pull request that touches the front end, including passing ones — a green comment builds the habit of reading it.
  4. Post-deploy field checks are recorded and can be read back per release.
  5. Overrides are countable, with a reason attached to each.

What the gate cannot catch

Being explicit about the blind spots is what stops a green check from being mistaken for a guarantee.

Third-party changes. A vendor who ships a heavier tag next Tuesday will not fail any pull request, because no pull request happened. Only field alerting sees it.

Traffic-mix effects. A campaign that brings low-tier mobile traffic moves every percentile with no code change at all. The lab has one device and cannot represent that.

Session-accumulated cost. A lab run measures a fresh page load. The INP degradation that appears after five route changes, described in Soft Navigations & SPA Metrics, is invisible to it.

Personalised and authenticated paths. If the gate tests the anonymous page and most traffic is logged in, the two are different applications.

Each of those is a reason the field half of the two-stage design exists. The lab gate catches what a change introduced; the field check catches what the world introduced. A programme with only the first is fast and blind, and a programme with only the second is accurate and slow.

What the gate caught over a quarterThe deterministic check is both more accurate and more actionable. The metric check earns its place, but its flake share is the number to keep under control.What the gate caught over a quarterOutcome of every failing check, 340 pull requestsResource budget71%14%12%Lab metric budget52%18%13%17%Real regression fixedOverridden deliberatelyBudget was wrongFlake
The deterministic check is both more accurate and more actionable. The metric check earns its place, but its flake share is the number to keep under control.

Edge cases and gotchas

  • Shared CI runners are noisy. Neighbouring jobs change the numbers. A dedicated runner or a hosted measurement service is the difference between a gate and a lottery.
  • Preview URLs behave differently. Cold caches, different CDN configuration, blocked third parties. Know the differences or you will chase them repeatedly.
  • Blocked third-party domains in CI. A corporate network that blocks analytics makes the preview lighter than production, so the gate under-reports.
  • Gating on the composite score. It moves with tool versions. Gate on individual metrics with explicit budgets.
  • Authenticated routes. If most of your traffic is logged in and the gate tests the anonymous page, it is measuring a different application.
  • A gate that never fails. Either the budget is too loose or the check is not running. Both are worth an hour to find out.

FAQ

Should a failing performance check block a merge?

The resource budget, yes — it is deterministic and the author can fix it in the same pull request. The lab metric check, yes with an override label. The field check, no: it lags and is subject to traffic mix.

How many runs are enough?

Five, taking the median. Fewer and run-to-run variance dominates; more rarely changes the verdict and costs CI minutes.

What about routes CI cannot reach?

Authenticated or checkout routes usually need a fixture login. If that is not feasible, exclude them from the gate and rely on field alerting, and say so explicitly rather than leaving a silent gap.

How do I stop people disabling the check?

Give them a legitimate override that is visible in review and logged. An override used three times on one route is a signal the budget is wrong — which is information you only get if the escape hatch exists.