Defending a Null

Every part so far has been about detecting an effect. This one is about the harder job of showing there is none, which comes up more often than you would think. Does the guide help pupils who are resitting the exam as much as it helps pupils sitting it for the first time? The twelve-school trial's P5 expected "about the same". Does the deluxe guide, with its video worked solutions, earn its extra cost over the plain one? The headteacher would be glad to hear that it does not, since the plain one is cheaper. And the teacher's plan for next year's trial has, as its most likely honest headline, that the cheaper guide does just as well. Each of those is a claim of absence, and the machinery of Parts 4 to 7 cannot make it.

The asymmetry

Recall the logic. We assume the null, the claim that the guide does nothing, and ask how surprising the data would be if that were so. If very surprising, we reject the null. If not, we keep it. Keeping it is not confirming it, for the reason Part 3 gave: not guilty is not innocent. The test was built to detect, and a detector that did not go off is a detector that did not go off.

P5 is the cleanest example in the twelve-school trial. The guide's benefit for pupils resitting the exam, minus its benefit for pupils sitting it for the first time, came to −0.7 marks, with a 95 per cent confidence interval, the range of gaps the data cannot rule out, running from −13.7 to +12.7, and a p-value, corrected for the six predictions, of 0.9989. The gap is as close to zero as a gap gets. And the teacher's write-up, correctly, did not say "the guide helps resit pupils and first-timers equally". It said that the prediction was not supported, and explained why in one line: support would have needed an interval hugging zero, not just a test that failed to find a difference. The interval was 26 marks wide. A 13-mark benefit either way was still on the table. A shrug at p = 0.9989 is still a shrug.

Absence of evidence, evidence of absence

The phrase is a cliché because it is exactly right. "We found no significant effect" is absence of evidence: the detector did not go off, and the interval tells you how insensitive the detector was. "The effect, if any, is smaller than five marks" is evidence of absence: a positive claim, with a number in it, that a study can support or fail to support like any other.

The difference between the two intervals in the figure is not their centre. Both sit on −0.7. It is their width, and width is bought with pupils. The lower interval is what P5 would have looked like with roughly four times the pupils, and it supports the stronger claim because it fits inside a band. Which raises the question of where the band came from.

The smallest effect worth caring about

To say "no meaningful difference" you have to say what "meaningful" means, in the outcome's units, before you see the data. This number is the smallest effect size of interest, and choosing it is a judgement about the world rather than a statistical calculation. On an exam marked out of 100, five marks is about what one question is worth, a quarter of the distance from the pass mark to a top grade, and a third of the spread between two pupils who had exactly the same preparation. A guide whose effect is inside ±5 is, for every practical purpose, a guide that does nothing, and not one a headteacher would buy for every pupil. So five is a defensible bound, and the point is not that it is right but that it is written down first.

Write it down first because otherwise the bound will be chosen to fit the interval. If the interval had come out at −8 to +7, a bound of 10 would make the null look defended, and a reader has no way of knowing whether 10 was the plan or the rescue. This is the single strongest argument for the preregistration of Part 10, and it is why the teacher's plan for next year lodges its bounds with a third party before the first pupil is handed a guide.

The equivalence test

With a bound in hand, the claim "the effect is within ±5 marks" becomes testable, and the test is called an equivalence test. The standard form is two one-sided tests, usually written TOST, and the name describes it exactly.

Test one: is the effect significantly greater than −5? Test two: is the effect significantly less than +5? If both pass at the 5 per cent level, the effect is inside the band, and the null has been defended: not "we could not find a difference" but "the difference, if any, is smaller than five marks".

There is a picture that makes the procedure obvious. Two one-sided tests at 5 per cent each are the same thing as checking whether a 90 per cent confidence interval lies entirely inside the band from −5 to +5. Ninety, not ninety-five, because each one-sided test spends its 5 per cent on one end. So the recipe is: build the 90 per cent interval by the bootstrap of Part 6, which resamples the pupils you have to see how far the gap could wander, draw the band, and look.

The four things that can happen, on one axis:

The last row is the one that costs the most, because it means the pupils' fortnight was spent and nothing was learned, and it is the row that a well-designed null study is built to avoid. (Part 15 gives the same band a probability instead of a verdict, which handles the middle cases more gracefully.)

Power: the chance of seeing what is there

The reason P5 landed in the last row was not bad luck. It was arithmetic that could have been done before the trial, and the concept is statistical power: the probability that a study will detect an effect of a given size, if that effect is really there.

Power depends on three things. The size of the effect you are looking for, the spread between pupils who had the same preparation, and the number of pupils. Bigger effects, less spread and more pupils all raise it. The convention is to aim for 80 per cent, meaning that if the effect exists, four studies in five will find it and one in five will miss it, a rate of misses that researchers have decided to live with.

Here is what it looks like on the scale of the twenty-four pupils, with about 15 marks of spread between pupils, for the ordinary two-sided test at the 0.05 line.

Six pupils a side, the size of one cell in the twenty-four, can reliably detect only a gap of 27 marks, more than twice the guide's real effect. The twelve-school trial had 192 pupils a side in its main comparison, the plain guide against no guide, and could see a 4-mark gap, which is why P1's 12 marks came through with room to spare. P6, with about 28 pupils a side, could reliably see 11 marks and was looking at 15, which is a coin toss made slightly generous, and it came out just on the wrong side of the line.

The figures come from the standard power calculation for a two-sample test, and a rule of thumb gets close for anything but the smallest samples: the smallest detectable gap is about 2.8 times the spread divided by the square root of half the pupils per condition. Any statistics package will give it, and a power analysis in a methods section is just this calculation run in reverse: decide the gap you need to see, and it tells you how many pupils to recruit. For a 12-mark gap with 15 marks of spread the answer is 26 pupils a side.

How many pupils to defend a null

For an equivalence test the question is how many pupils make the 90 per cent interval narrow enough to fit in the band. With 15 marks of spread and a band of ±5, the interval's half-width is about 1.645 × 15 × the square root of 2 divided by the pupils per condition, and setting that to 5 gives about 50 pupils a side. That is the number at which the interval on average just fits, and it is a trap, because the interval is centred on a measured gap that itself wanders a few marks either side of zero. Simulate it and only one null study in twenty-five comes out equivalent at 50 pupils a side, and one in four at 70. For the usual 80 per cent power the arithmetic says about 155 pupils a side, three times the naive number, and that is the figure a power analysis for a null-result study has to produce before the pupils are recruited.

Now P5 again. There were 144 resit pupils across the schools that had them, which sounds like plenty, but P5 was a difference of differences: the guide's gap among the resit pupils minus the guide's gap among the first-timers. That is four groups and two subtractions, and every subtraction adds noise: the interval for a difference of two gaps is about twice as wide as the interval for one gap on the same number of pupils. Twice the width means four times the pupils to get back to the same precision, and the trial did not have them. A power analysis done in advance would have said so, and the honest report of an inconclusive result is what the teacher's write-up printed instead. Her plan for next year does the analysis first.

Try it: bounding a null

Set the band, the spread, the true gap and the pupils per condition. Draw a study simulates one study, builds its 90 per cent interval, and gives the verdict. Draw 20 shows how often a study of this size reaches each verdict, which is its power.

Two things to try. With the true gap at 0, 15 marks of spread and 12 pupils a side, press Draw 20 and read the tally: nearly every study is inconclusive, which is P5's situation. Slide the pupils up to 70 and draw 20 again: only about a quarter come out equivalent, even though the interval "fits on average" at 50. Slide to 160 and most do. Then set the true gap to 12 and pupils to 6, and count how many studies find it. That fraction is the power, and it will be poor.

In Python

The equivalence test is the interval rule, four lines long. The simulation is the honest way to find a sample size: run the study you are planning a few thousand times on a true null and count how often it comes out equivalent. The power calculation gives the other numbers in this part.

import numpy as np
from scipy import stats
from statsmodels.stats.power import TTestIndPower

# Equivalence test (TOST) by the interval rule: does the 90% interval fit inside the band?
def tost(a, b, bound):
    gap = a.mean() - b.mean()
    se = np.sqrt(a.var(ddof=1) / len(a) + b.var(ddof=1) / len(b))
    lo, hi = gap + stats.t.ppf([0.05, 0.95], len(a) + len(b) - 2) * se   # 90% interval = two one-sided tests
    return ("equivalent" if -bound <= lo and hi <= bound else
            "inconclusive" if lo <= 0 <= hi else "different")

# Simulate a true null (the guide does nothing, 15 marks of spread between pupils) many times at two sample sizes
rng = np.random.default_rng(20260703)
for n in (12, 70):
    verdicts = [tost(rng.normal(62, 15, n), rng.normal(62, 15, n), bound=5) for _ in range(2000)]
    eq = verdicts.count("equivalent") / 2000; inc = verdicts.count("inconclusive") / 2000
    print(f"{n:>2} pupils a side, band ±5: equivalent {eq:.0%}, inconclusive {inc:.0%}, wrong {1 - eq - inc:.0%}")

# Power: how many pupils to detect a gap of 12 marks with 15 marks of spread, 80% of the time
power = TTestIndPower()
n = power.solve_power(effect_size=12 / 15, alpha=0.05, power=0.8)
print(f"pupils per condition for 80% power on a 12-mark gap: {n:.0f}")

# The other way round: the smallest gap a given number of pupils can reliably detect
for n in (6, 12, 24, 48, 100, 192):
    d = power.solve_power(nobs1=n, alpha=0.05, power=0.8)
    print(f"{n:>3} pupils a side detect a gap of about {15 * d:.0f} marks")

which prints:

12 pupils a side, band ±5: equivalent 0%, inconclusive 90%, wrong 10%
70 pupils a side, band ±5: equivalent 25%, inconclusive 65%, wrong 9%
pupils per condition for 80% power on a 12-mark gap: 26
  6 pupils a side detect a gap of about 27 marks
 12 pupils a side detect a gap of about 18 marks
 24 pupils a side detect a gap of about 12 marks
 48 pupils a side detect a gap of about 9 marks
100 pupils a side detect a gap of about 6 marks
192 pupils a side detect a gap of about 4 marks

Where this leaves us

The ordinary test detects. It cannot certify absence, because a detector that stayed silent may have been too weak to hear. To claim "no meaningful effect" you must say in advance what meaningful means, in the outcome's units, and then show that the whole 90 per cent interval fits inside that band: two one-sided tests, or one picture. How many pupils that takes is a calculation, power analysis, that can and should be done before a single guide is handed out, and the twelve-school trial's one inconclusive result is what happens when it is not. The next two parts leave the arithmetic and turn to the two things that decide whether any of it is believed: how the study was built, and when its choices were made.

Next: Designed to Be Believed: names drawn from a hat versus volunteers, confounds and the one School C's year admitted to, markers who do not know who had the guide, a mark scheme written before the exam, the intention-to-treat rule borrowed from medicine, publishing enough for others to get the same numbers, and how to read a kappa of 0.973.