Resampling Beyond the Interval

The first ten parts used two resampling methods for two jobs: shuffling labels to get a p-value (Part 4) and resampling pupils to get an interval (Part 6). Write-ups use the same two words, permutation and bootstrap, in more ways than that, and this part collects the ones you will meet. The bootstrap as a test. The paired bootstrap, which is the right comparison for two conditions measured on the same seven topics of one exam paper. The bootstrap for statistics that have no formula. And the jackknife, the bootstrap's older and simpler relative, which answers a question of its own.

The bootstrap as a test

Part 6 built a confidence interval, the range of gaps that the data cannot rule out, by resampling each condition's pupils with replacement. If that interval excludes zero, the gap is significant at the matching level, so the bootstrap already gives you a test of a sort. There is also a direct version, and it is worth seeing because it makes the difference between a bootstrap test and a permutation test precise.

A permutation test asks: if the two conditions were the same in every way, how often would shuffling produce this gap? It builds the null, the imaginary world in which the guide does nothing, by mixing the two groups into one pool.

A bootstrap test asks something slightly narrower: if the two conditions had the same mean but kept their own spreads, how often would resampling produce this gap? It builds the null differently. Slide each group so that its mean sits on the common mean of all twenty-four pupils, leaving every mark's distance from its own group's mean untouched. Now the null is true by construction, each group keeps its own shape, and resampling from the two shifted groups shows what gaps a world with equal means would produce.

The two numbers differ because the groups differ in spread, 15.5 marks against 17.4, and the bootstrap null keeps that while the permutation null erases it. Neither is wrong. They answer different questions, and for most studies the permutation test's question, "are these two conditions distinguishable at all", is the one being asked, which is why it is the default and why the twelve-school trial used it. The bootstrap test earns its place when the spreads clearly differ and the mean is the only thing you are making a claim about, or when the statistic is something the permutation test cannot handle, which is the subject of the next section but one.

The paired bootstrap

Here is a design that comes up constantly in classroom studies and that the methods so far have handled badly. The exam paper has seven topics, and the markers gave a separate mark for each topic. For each topic the teacher worked out the average mark of the twelve pupils with the guide and the average of the twelve without. Fourteen numbers, not twenty-four, and they come in pairs.

Look down the difference column: every topic gained, by between 9 and 15 marks. The topics themselves vary far more, from geometry at 52 without the guide to number at 75. Easy topics score high in both groups and hard topics score low in both. Treat the fourteen numbers as two independent groups of seven, resample each, and the topic-to-topic variation floods the interval: +2.9 to +19.1, a span of sixteen marks that only just clears zero and would make the guide's effect look far less certain than it is. That is the wrong analysis, because the two groups are not independent. Each topic's "with" average and "without" average share the topic's difficulty, and the difficulty cancels in the difference.

The paired bootstrap resamples the unit that the pairs are built on, the topic. Draw seven topics with replacement, take the mean of their differences, repeat. The interval is +9.7 to +12.7: five times narrower, from exactly the same numbers, because the pairing removed a source of noise that was never about the guide. This is the resampling version of the paired t-test of Part 12, and the general rule behind both is the one from Part 4: resample or shuffle the unit that the design made independent, and when the numbers come in pairs, keep the pairs together. The twelve-school trial's stratification by school and topic is the same rule at a larger scale.

Anything you can compute, you can bootstrap

The formula intervals of Part 6 exist for means, and with some effort for a few other things. The bootstrap does not care what the statistic is. Compute it on a resample, repeat, read the tails. Three that come up in write-ups of studies like the teacher's:

A ratio of medians. The median is the middle value when the numbers are put in order, and the twelve-school trial reported that pupils with the guide revised for 2.35 times as long as pupils without, medians of 615 minutes against 262 over the fortnight. (P3's ×2.2 is the same gap summarised a different way, as the average multiplier on a log scale, which is pulled less by the keenest revisers.) A ratio of medians has no textbook interval. Resample each group, take the ratio of the two resampled medians, and 10,000 repeats give an interval directly. On the twenty-four pupils' own revision logs, twelve a side, the multiplier is ×2.38 with an interval of ×2.16 to ×2.69.

A correlation. Do the pupils who revised for longest get the highest marks? Part 13 computes the correlation, a number for how consistently two things move together, and an interval for it comes from resampling the pairs (pupil, minutes, mark) and recomputing.

A difference of differences. P5's statistic was the guide's gap among pupils resitting the exam minus its gap among pupils sitting it for the first time. Four groups, resampled within each cell, recombined, repeated. The twelve-school trial's interval for it, −13.7 to +12.7, was built exactly this way.

The one rule is to resample at the level of the design: pairs stay together, cells stay separate, and a statistic computed on a resample is computed the same way as on the original.

The jackknife

Before the bootstrap there was the jackknife, from the 1950s, and it is still the right method for one question: which single observation is doing the work? Leave each observation out in turn, recompute the statistic without it, and look at how far each omission moves the answer.

On the twelve families' tutoring bills of Part 2, leaving out the big spender moves the mean from £577 to £346. Leaving out any other family moves it by £36 at most. That is a numerical way of saying what the picture said: one family owns the mean. A write-up that says "the result survived leaving any single pupil out" has done this, and it is a cheap, convincing check for a small study. The jackknife also gives a standard error, the typical distance between an estimate and the truth, and for smooth statistics like means it agrees with the bootstrap's, but for anything with a step in it, medians and other cut-points of the data (quantiles, the value below which a given fraction of the pupils fall) especially, it can be badly wrong, which is the main reason the bootstrap replaced it.

When resampling lets you down

The bootstrap is not magic, and three of its failures are worth knowing by name.

Too few pupils. Resampling six pupils with replacement produces only a few hundred distinct resamples, and an interval drawn from them is jagged and too narrow. Below about eight or ten a side, prefer a permutation test, which is exact at any size, and treat any interval as rough.

The extremes. The bootstrap cannot estimate the uncertainty of a maximum or a minimum, because a resample can never contain a value larger than the largest one you have. "The best pupil scored 95" has no bootstrap interval worth reporting.

Heavy tails. With a distribution like the tutoring bills, where one family in twelve can spend ten times what the others spend, the resampled means depend on how many times the big spender happens to be drawn, and the interval for the mean is unstable. This is the same lesson as Part 2: report the median, bootstrap the median, and the problem goes away.

The twelve-school trial stayed inside these limits: forty-eight or more pupils in every condition, means of marks out of 100, which cannot run away, and medians for the minutes of revision.

In Python

Each of the four ideas above is a few lines once you have the resampling loop from Part 6. The paired example uses the seven-topic table, the ratio example uses the twenty-four pupils' minutes of revision, and the jackknife runs on Part 2's tutoring bills.

import numpy as np

with_guide    = np.array([88, 91, 79, 84, 95, 86, 62, 58, 71, 49, 66, 55])
without_guide = np.array([78, 84, 70, 75, 89, 62, 46, 53, 40, 58, 49, 37])
observed = with_guide.mean() - without_guide.mean()
rng = np.random.default_rng(20260703)

# 1. The bootstrap test: move both groups onto a common mean, so the null is true,
#    then resample and ask how often a gap as large as the real one appears
grand = np.concatenate([with_guide, without_guide]).mean()
null_with, null_without = with_guide - with_guide.mean() + grand, without_guide - without_guide.mean() + grand
gaps = np.array([rng.choice(null_with, 12).mean() - rng.choice(null_without, 12).mean() for _ in range(10_000)])
print(f"bootstrap test, pooled: p = {np.mean(np.abs(gaps) >= abs(observed)):.3f}   (permutation gave 0.093)")

# 2. The paired bootstrap: seven topics, each averaged under both conditions. Resample topics, not pupils.
topics = ["fractions", "ratio", "algebra", "percentages", "statistics", "geometry", "number"]
with_by_topic    = np.array([81, 74, 66, 79, 70, 62, 84])
without_by_topic = np.array([70, 61, 57, 68, 55, 52, 75])
diff = with_by_topic - without_by_topic
paired = np.array([rng.choice(diff, 7).mean() for _ in range(10_000)])
unpaired = np.array([rng.choice(with_by_topic, 7).mean() - rng.choice(without_by_topic, 7).mean() for _ in range(10_000)])
print(f"gap {diff.mean():+.1f}: paired interval {np.percentile(paired, 2.5):+.1f} to {np.percentile(paired, 97.5):+.1f}, "
      f"unpaired {np.percentile(unpaired, 2.5):+.1f} to {np.percentile(unpaired, 97.5):+.1f}")

# 3. Anything you can compute, you can bootstrap: how many times longer pupils with the guide revised, as a ratio of medians
mins_with    = np.array([612, 540, 720, 655, 590, 810, 575, 640, 700, 530, 615, 690])   # minutes of revision in the fortnight
mins_without = np.array([250, 300, 262, 240, 275, 310, 255, 290, 230, 270, 265, 245])
ratio = np.median(mins_with) / np.median(mins_without)
boots = [np.median(rng.choice(mins_with, 12)) / np.median(rng.choice(mins_without, 12)) for _ in range(10_000)]
print(f"revision multiplier x{ratio:.2f}, 95% interval x{np.percentile(boots, 2.5):.2f} to x{np.percentile(boots, 97.5):.2f}")

# 4. The jackknife: leave each family out in turn and see how far the estimate moves
bills = np.array([181, 204, 238, 257, 269, 291, 312, 348, 402, 517, 784, 3120])   # pounds spent on tutoring
loo = np.array([np.delete(bills, i).mean() for i in range(12)])
worst = np.argmax(np.abs(loo - bills.mean()))
print(f"mean bill {bills.mean():.0f}; leaving out family {worst + 1} (bill {bills[worst]}) moves it to {loo[worst]:.0f}; "
      f"no other family moves it by more than {np.delete(np.abs(loo - bills.mean()), worst).max():.0f}")

which prints:

bootstrap test, pooled: p = 0.065   (permutation gave 0.093)
gap +11.1: paired interval +9.7 to +12.7, unpaired +2.9 to +19.1
revision multiplier x2.38, 95% interval x2.16 to x2.69
mean bill 577; leaving out family 12 (bill 3120) moves it to 346; no other family moves it by more than 36

Where this leaves us

Permutation and bootstrap are two ways of asking "what else could the data have looked like", and they build different imaginary worlds: one where the labels are interchangeable, one where each group keeps its shape but its mean is pinned. The bootstrap's real power is that it works for any statistic, a ratio, a correlation, a difference of differences, as long as the resampling follows the design, and the paired case is where following the design matters most: resample the topics as pairs, not the fourteen numbers as two loose groups, and an interval shrinks fivefold from the same numbers. The next part turns to the tests that predate all of this, which still fill most results sections, and shows which resampling test each one is a formula for.

Next: The Classical Toolbox: Student's t and Welch's t, the paired t-test, Mann-Whitney and Wilcoxon, chi-square against Fisher, ANOVA and Kruskal-Wallis, with a table that says which one to reach for and which resampling test it corresponds to.