The Claim and Its Shadow

The twenty-four pupils of Part 1 gave a gap of 12 marks. Before asking whether chance could produce it, we have to say what we think produced it, and say it precisely enough that the data could prove us wrong. That statement is a hypothesis, and the discipline of writing one down before looking is most of what separates a study from an anecdote.

This part is about how to write one, what its shadow is, and what it means for a hypothesis to be "supported". It uses the six predictions from the twelve-school trial, because they were written before any analysis, and because three of them were not supported, which makes them better teaching material than six that were.

Four ingredients

A hypothesis is a bet, and like any bet it has to be specific enough to be settled. Here is the first of the trial's six, and its four parts.

Pupils given the guide for the fortnight before the exam will get a higher mark on the end-of-year maths exam than pupils not given it, across all twelve schools in the trial.

Here are three hypotheses that look fine and are not.

"The guide is useful." No comparison, no outcome, no direction. It cannot lose, which means it cannot win either.

"The guide improves results." Improves what? At School C the guide lifted the pass rate from 5 of 18 to 16 of 18, and the pupils who had it spent far longer revising to get there. Whether "improves" is true depends entirely on which outcome you name.

"The deluxe guide will change the geometry mark." Four of the schools also tried a deluxe version of the guide, with video worked solutions, alongside the plain one. This sentence has a comparison and an outcome, but no direction, and "change" is not what anyone believes. The trial's actual P4 said "higher", and then, as we will see, had to report that it could not confirm it.

The shadow

Every hypothesis casts a shadow: the statement that nothing is going on. For the guide hypothesis, the shadow is "the guide makes no difference to the exam mark". This is the null hypothesis, and the odd, important fact about statistics is that it is the shadow that gets tested, not the claim.

Why test the thing you do not believe? Because the shadow is the only one of the two whose consequences can be calculated. If the guide makes no difference, then the label "with" or "without" on each pupil is just a label, and the marks would have come out the same with the labels swapped. That is a precise, mechanical statement, and Part 4 turns it into a procedure: swap the labels, thousands of times, and see how big a gap swapping alone produces. The claim, by contrast, says only that the guide helps by some amount, and "some amount" does not tell you what the data should look like.

So the logic runs backwards from what intuition expects:

The procedure has three steps. First, assume the shadow: the guide does nothing. Second, work out how surprising the observed gap of 12 would be if that were true. Third, if it would be very surprising, abandon the shadow, and by elimination the claim stands. If it would not be surprising, keep the shadow, and the claim is not supported.

Note the wording of the last step. Keeping the shadow is not the same as proving it. A courtroom is the standard analogy and it is a good one.

A verdict of not guilty leaves open that the defendant did it and the case was weak. A hypothesis that is not supported leaves open that the guide helps and the study was too small to see it. The twelve-school trial's P4 is exactly this: pupils with the deluxe guide scored 6.9 marks higher on the geometry questions, and the trial could not rule out chance to the required standard, so P4 was reported as not supported rather than as "the videos do not help". Part 8 is about what it takes to say the stronger thing.

The alternative, and which way it points

The claim itself, once the shadow has been rejected, is called the alternative hypothesis. It comes in two shapes.

A one-sided alternative names a direction: the guide raises the mark. A two-sided alternative says only that the mark changes, up or down. The difference sounds like a technicality and is not, because a one-sided test only counts surprise in the named direction, which makes it easier to pass, and that ease is open to abuse: pick the direction after seeing which way the data went and a one-sided test flatters you.

The twelve-school trial expected a direction for five of its six predictions and still tested all of them two-sided. That is the cautious convention and the one I would recommend. A guide can hurt as well as help, if it eats the fortnight a pupil would otherwise have spent on their weakest topic, and being surprised by a drop should count as being surprised. When you read a write-up, the phrase "two-sided" or "two-tailed" beside a p-value (the number that says how surprising the gap would be if the shadow were true, which Part 5 defines properly) is a small mark of good faith.

Six bets, written out

Here are the six primary hypotheses from the twelve-school trial, each reduced to its four ingredients, with what happened. The effect sizes and the corrected p-values will be explained in Parts 5 to 7. For now, read the first four columns as bets placed before the data existed, and the last as how they were settled.

Three things to notice.

P3 is a hypothesis about a cost, and it was expected to come out "higher". Supported, here, means "yes, the guide costs the pupils time, as we said it would". A hypothesis does not have to be good news.

P5 is a hypothesis whose expected answer was no difference: the guide should help a pupil resitting the exam about as much as a pupil sitting it for the first time. That is a legitimate thing to predict, but it cannot be settled by the machinery above, because the machinery only ever rejects the shadow, and here the shadow is the claim. The trial reported it plainly: the gap was near zero, and the interval around it was so wide that it would have been near zero whatever the truth was. Part 8 covers how to make a "no difference" hypothesis testable, which is how P5 would have to be written next time.

P6 has the largest effect of the six, 15 marks, and was not supported, because it rested on one school's extra algebra paper with about 28 pupils a side. That is the sample-size lesson of Part 1 showing up in a real table: a big gap on few pupils is weaker evidence than a modest gap on many.

Primary, secondary, pre-specified

The six above are primary hypotheses, the trial's headline bets. Because there were six of them, chance had six tries at producing an impressive-looking gap, and Part 7 explains the correction the trial applied for that. The trial also asked a further set of questions after the fact, things like "does the free one-page revision sheet alone get most of the guide's benefit?", and reported them as secondary contrasts: exploratory, uncorrected, and labelled as such, so that no reader mistakes them for bets that were placed in advance.

Pre-specified means the hypotheses were written, with all four ingredients and the test to be used, before the analysis was run. The twelve-school trial's write-up says the six were "decided in advance", and then adds a sentence I would encourage anyone to copy: that the plan was written down and dated before the exam, but kept in a private document rather than lodged with anyone outside the schools. Part 10 is about that distinction and why it matters more for some studies than for others.

Try it: the hypothesis builder

Pick the four ingredients and the box writes the hypothesis, its shadow, and what each verdict would mean. The presets are the six from the trial.

Press P5 and read what the box says about a claim whose expected answer is "the same". Then set the direction to "different, either way" for P1 and notice that the verdict text changes: a two-sided claim is supported by a surprise in either direction, which is why it is the safer default.

In Python

A hypothesis is data, and writing the six down as records makes it obvious when one is missing an ingredient, and which one cannot be settled by the ordinary test.

# A hypothesis is data: write the six down with all four ingredients, then check nothing is missing
hypotheses = [
    dict(id="P1", compare=("the guide", "no guide"), outcome="exam mark", direction="higher", scope="all twelve schools"),
    dict(id="P2", compare=("the guide", "no guide"), outcome="pass rate", direction="higher", scope="all twelve schools"),
    dict(id="P3", compare=("the guide", "no guide"), outcome="minutes of revision (log scale)", direction="higher", scope="all twelve schools"),
    dict(id="P4", compare=("the deluxe guide", "the plain guide"), outcome="geometry mark", direction="higher", scope="the four schools with both versions"),
    dict(id="P5", compare=("the guide's benefit for resit pupils", "its benefit for first-timers"), outcome="exam mark", direction="same", scope="the schools with resit pupils"),
    dict(id="P6", compare=("the guide", "no guide"), outcome="algebra paper mark", direction="higher", scope="one school, about 28 pupils a side"),
]
for h in hypotheses:
    a, b = h["compare"]
    null = f"{a} makes no difference to {h['outcome']} compared with {b}"
    testable = "needs an equivalence bound (Part 8)" if h["direction"] == "same" else "two-sided permutation test"
    print(f"{h['id']}: {a} vs {b}, {h['outcome']}, expected {h['direction']}. Null: {null}. Test: {testable}")

which prints:

P1: the guide vs no guide, exam mark, expected higher. Null: the guide makes no difference to exam mark compared with no guide. Test: two-sided permutation test
P2: the guide vs no guide, pass rate, expected higher. Null: the guide makes no difference to pass rate compared with no guide. Test: two-sided permutation test
P3: the guide vs no guide, minutes of revision (log scale), expected higher. Null: the guide makes no difference to minutes of revision (log scale) compared with no guide. Test: two-sided permutation test
P4: the deluxe guide vs the plain guide, geometry mark, expected higher. Null: the deluxe guide makes no difference to geometry mark compared with the plain guide. Test: two-sided permutation test
P5: the guide's benefit for resit pupils vs its benefit for first-timers, exam mark, expected same. Null: the guide's benefit for resit pupils makes no difference to exam mark compared with its benefit for first-timers. Test: needs an equivalence bound (Part 8)
P6: the guide vs no guide, algebra paper mark, expected higher. Null: the guide makes no difference to algebra paper mark compared with no guide. Test: two-sided permutation test

Where this leaves us

A hypothesis is a bet with four parts: what is compared, on which outcome, in which direction, and over what scope. Its shadow, the null hypothesis, is the statement that the comparison makes no difference, and it is the shadow that gets tested, because only the shadow says exactly what the data should look like. "Supported" means the shadow was rejected. "Not supported" means it was not, and that is all it means. The next part does the rejecting, by taking the shadow at its word: if the labels on the twenty-four pupils do not matter, swap them and see.

Next: Shuffle the Labels: the permutation test, one shuffle at a time. Why the gap of 12 in the twenty-four pupils is not surprising when you shuffle everything, and very surprising when you shuffle within each school, and what that says about how the twelve-school trial ran its tests.