Calculating Confidence Intervals for a Sampled p75

Every percentile computed from sampled data is an estimate, and an estimate without an interval invites two errors: acting on noise, and dismissing a real regression as noise. This guide sits under RUM Data Sampling Strategies and covers two ways to attach an interval to a p75 — an exact order-statistic method that needs no simulation, and a bootstrap for the cases where it does not apply — plus the sample-size floors those methods imply for cohort reporting.

Prerequisites

Raw or lightly aggregated event data, not pre-computed percentiles. You cannot recover uncertainty from a stored p75.
The sampling rate that produced the data, recorded per beacon or per configuration version.
A cohort definition you want the interval for — a route, a device tier, or a route crossed with a device tier.

How to compute it

Step 1 — Use the order-statistic method first

A percentile is an order statistic, and the number of observations below a quantile follows a binomial distribution. That gives an exact interval with no simulation: find the ranks that bound the true quantile with the confidence you want, and read the values at those ranks.

// p75 interval from the sorted sample. No resampling required.
export function p75Interval(values, z = 1.96) {
  const xs = values.slice().sort((a, b) => a - b);
  const n = xs.length;
  if (n < 30) return null;                    // too few to say anything useful
  const q = 0.75;
  const centre = n * q;
  const spread = z * Math.sqrt(n * q * (1 - q));

  const lowRank = Math.max(0, Math.floor(centre - spread));
  const highRank = Math.min(n - 1, Math.ceil(centre + spread));

  return {
    estimate: xs[Math.min(n - 1, Math.floor(centre))],
    lower: xs[lowRank],
    upper: xs[highRank],
    n,
  };
}

The width of that interval is what tells you whether a cohort is worth reporting. With 1,000 samples the ranks are roughly 723 and 777 — a narrow band on most distributions. With 100 samples they are 66 and 84, which on a heavily right-skewed metric can span a second or more.

Detecting a 5% regression needs an interval narrower than 5% — which puts the floor somewhere between 1,000 and 5,000 samples for this distribution.

Step 2 — Bootstrap when the assumptions do not hold

The order-statistic method assumes independent observations. Sessions are not independent — one user generating twenty page views contributes correlated values — and stratified or weighted sampling breaks it further. A bootstrap handles both, at the cost of computation.

// Cluster bootstrap: resample sessions, not events, so within-session
// correlation is preserved rather than averaged away.
export function bootstrapP75(sessions, iterations = 1000) {
  const p75 = (xs) => {
    const s = xs.slice().sort((a, b) => a - b);
    return s[Math.floor(s.length * 0.75)];
  };
  const estimates = [];
  for (let i = 0; i < iterations; i++) {
    const draw = [];
    for (let j = 0; j < sessions.length; j++) {
      const session = sessions[Math.floor(Math.random() * sessions.length)];
      for (const v of session.values) draw.push(v);
    }
    estimates.push(p75(draw));
  }
  estimates.sort((a, b) => a - b);
  return {
    estimate: estimates[Math.floor(iterations * 0.5)],
    lower: estimates[Math.floor(iterations * 0.025)],
    upper: estimates[Math.floor(iterations * 0.975)],
    iterations,
  };
}

A thousand iterations is enough for a 95% interval; ten thousand changes the third decimal place and rarely the decision.

Step 3 — Compute it in the warehouse for routine reporting

Running a bootstrap over every cohort nightly is expensive. For routine dashboards the order-statistic approach can be expressed directly in SQL.

-- p75 with an approximate 95% interval, per route, in one pass.
WITH ranked AS (
  SELECT
    route,
    value,
    row_number() OVER (PARTITION BY route ORDER BY value) AS rn,
    count() OVER (PARTITION BY route)                     AS n
  FROM rum_events
  WHERE metric = 'LCP' AND ts >= now() - INTERVAL 7 DAY
)
SELECT
  route,
  any(n)                                                             AS samples,
  maxIf(value, rn = toUInt32(n * 0.75))                              AS p75,
  maxIf(value, rn = toUInt32(greatest(1, n * 0.75 - 1.96 * sqrt(n * 0.75 * 0.25))))  AS p75_lower,
  maxIf(value, rn = toUInt32(least(n,  n * 0.75 + 1.96 * sqrt(n * 0.75 * 0.25))))    AS p75_upper
FROM ranked
GROUP BY route
HAVING samples >= 300
ORDER BY p75 DESC;

Step 4 — Correct for the sampling rate where it varies

If different cohorts are sampled at different rates — the usual outcome of stratified oversampling — the raw sample is not representative and the percentile must be computed on reweighted data.

// Weighted p75: each observation counts for 1 / samplingRate sessions.
export function weightedP75(rows) {
  const sorted = rows.slice().sort((a, b) => a.value - b.value);
  const total = sorted.reduce((s, r) => s + 1 / r.samplingRate, 0);
  let cumulative = 0;
  for (const r of sorted) {
    cumulative += 1 / r.samplingRate;
    if (cumulative >= total * 0.75) return r.value;
  }
  return sorted.at(-1)?.value ?? null;
}

Forgetting this step is the most common error in a stratified pipeline: oversampling a slow cohort and then computing an unweighted percentile makes the whole origin look worse than it is.

Step 5 — Turn the interval into a reporting rule

Samples in the cohort	Interval width (typical)	What to do with it
Under 300	Over 15%	Do not display a percentile at all
300–1,000	7–15%	Display with the interval; do not alert
1,000–5,000	3–7%	Alert on changes above 20%
Over 5,000	Under 3%	Alert on changes above 10%

Those thresholds are what connect this arithmetic to the alerting rules in Alerting & Regression Detection: a rule that fires on a 20% change needs an interval comfortably narrower than 20%, or it is firing on its own noise.

Step 6 — Show the uncertainty on the chart

A percentile plotted as a bare line implies a precision it does not have. Plot the band, or at minimum annotate the sample count on every panel. The single most effective change is putting the sample count next to the number: most bad decisions from RUM dashboards trace back to a cell nobody realised had forty observations in it.

Every apparent day-to-day movement in the middle line sits inside the band. Nothing happened in these ten days, and without the band three people would have investigated.

Using the interval to size a sampling decision

The arithmetic runs in the other direction too, and that is where it earns its keep in an architecture review. Given a change you want to be able to detect, the interval tells you how many samples you need, and therefore what sampling rate a cohort can tolerate.

Work backwards in three steps. Decide the smallest change worth detecting — 10% is a reasonable default for a load metric, since anything smaller is rarely worth a deploy. Read the sample size that gives an interval comfortably narrower than that, which for an LCP-shaped distribution is roughly one to five thousand. Then divide by the cohort’s natural traffic to get the maximum sampling rate you can apply while keeping the cohort useful.

A worked example makes the tension concrete. An origin with 30 million monthly sessions samples at 5% and is comfortable everywhere — until someone slices by route, device tier and country. A route carrying 2% of traffic, on the low device tier which is 20% of sessions, in a country that is 5% of traffic, has 30M × 0.02 × 0.2 × 0.05 × 0.05 = 300 sessions a month. That cell is below the floor and always will be at a flat 5%.

There are only three honest responses, and choosing between them is an architecture decision rather than a statistics one:

Oversample the cohort. Stratified sampling keeps small cohorts alive at the cost of a reweighting step at query time — the pattern is described in RUM Data Sampling Strategies.

Widen the window. A quarterly number for a small cohort is stable where a daily one is noise. State the window on the chart.

Do not slice that finely. Report the cell as part of a group and accept that you cannot see it individually.

What is not acceptable is displaying the 300-session cell next to the origin number with the same visual weight, which is what most dashboards do by default.

Verifying it works

The interval narrows as the square root of the sample size. Quadruple the samples and the width should halve; if it does not, the observations are not independent and you need session-level resampling.
The two methods agree on a large, well-behaved cohort. If the bootstrap interval is much wider, within-session correlation is significant.
Cohorts below the floor are suppressed, not displayed with a wide band that nobody reads.
Weighted percentiles are used wherever rates differ. Compute both once and confirm the difference is what you expect.
A simulated regression is detectable. Shift a copy of your data by 10% and check the intervals separate.

The order-statistic method is right for routine dashboards and wrong for exactly the pipelines that oversample small cohorts — which is most mature ones.

Edge cases and gotchas

Sessions, not events, are the sampling unit. Twenty page views from one user are not twenty independent observations, and treating them as such produces intervals that are too narrow.
Pre-aggregated percentiles cannot be interval-ed. If your pipeline stores a daily p75 and discards the raw data, this analysis is impossible after the fact. Keep quantile state, as covered in Building ClickHouse Materialized Views for Dashboards.
Approximate quantile algorithms add their own error. t-digest and similar structures are excellent, and their error is on top of the sampling error, not instead of it.
Heavy tails widen everything. LCP and INP distributions have long right tails, so intervals are wider than an equivalent sample from a normal distribution would suggest.
Multiple comparisons. Checking twenty cohorts at 95% confidence means one false positive per check by construction. Tighten the threshold or accept the rate deliberately.
The interval covers sampling error only. Instrumentation bias, delivery loss and bot traffic are not in it, and are frequently larger.

RUM Data Sampling Strategies — the sampling decisions that determine these sample sizes.
Alerting & Regression Detection — the rules these intervals set the floors for.
Device & Network Segmentation — where cohort sizes get small enough for this to bind.