The permutation test of Part 4 rests on one assumption: that under the null, any pupil could have been in either group. The intervals of Part 6 rest on the marks being honest measurements. Neither is guaranteed by the maths. Both are guaranteed, or not, by how the study was built, and no amount of statistics afterwards can repair a design that let something other than the guide decide the marks.

This part is the list of design choices that make a result believable, each attached to the place where the teacher's study got it right or wrong. School C's year is the honest starting point: a third school joined in a hurry, the trial there was run the way most studies are run the first time nobody is checking, and the methods section of the twelve-school trial is, in large part, a list of the things the teacher changed afterwards.

Volunteers or a hat

There is a way of running this study that feels natural, and that most teachers would choose: announce the guide, and hand a copy to every pupil who asks for one. The write-up would then have to say that the study was observational, with the groups chosen by the pupils rather than drawn by lot. What that means in practice is that the pupils who ask for the guide are the pupils who were going to revise anyway, and the gap at the end measures keenness as much as it measures the guide.

Even with names from a hat, the sorting is not finished, because anything else that differs between the two groups now travels with the condition. In a school the list is long.

Something that drifts How it becomes a confound
Time of day Pupils are sharper at nine in the morning than straight after lunch. If the pupils with the guide mostly sat in the morning, the morning sits in the guide's column
Which teacher Two classes, two teachers, and one of them explains fractions better. If one class had the guide and the other did not, the teacher travels with the guide
Which room A hot, echoing hall against a quiet classroom is worth a few marks on its own
The week of term A paper sat the week after half term meets fresher pupils than one sat at the end of a long term
Pupils who talk to each other Pupils with the guide lend it to friends, or pass on its tricks, so the "without" group is not quite without, and the gap shrinks for a reason that has nothing to do with the guide

Any of these can produce a gap between conditions that has nothing to do with the condition, and the shuffle of Part 4 cannot tell the difference, because from the data's point of view a morning effect and a guide effect look identical. School C's year named one of them. The exam there was sat in two sittings, and because the pupils with the guide had been gathered into one group for the fortnight, most of them sat in the morning and most of the others in the afternoon. The teacher had to write afterwards that time of day travelled with the guide, a residual confound, meaning a second difference between the groups that could explain the gap on its own, which is the right thing to do when it has already happened. The better thing is to prevent it.

Randomisation prevents it, provided it reaches everything that could drift. Draw the names from a hat before anything else is decided, and then do not let the two groups be separated again: the same sitting, the same hall, the same markers, and if there must be two sittings, draw those from the hat too. Then anything that drifts is equally likely to land in either condition, and the only systematic difference between the two groups is the thing you changed. That is what makes the labels exchangeable, meaning that any pupil could as easily have been in the other group, and it is what makes the permutation test's assumption true rather than hoped for.

The twelve-school trial went further and removed the drift where it could. At each school every pupil sat the same paper in the same hall at the same hour, taught by the same teachers through the same fortnight, and the guide was the single thing that differed between the groups at every one of the twelve schools. The write-up says so, with the timetables to prove it. When only one thing varies, only one thing can explain the result.

Blinding

Picture the teacher marking her own pupils' papers against the mark scheme, knowing, as she marks, which of them had the guide. She would not think that changed her marks. She also could not prove it, and neither could you, and the headteacher is entitled to assume it did.

The bias is not dishonesty. It is expectation. A marker who knows a pupil had the guide reads untidy working a little more charitably, gives a borderline method mark the benefit of the doubt, and does the opposite for the pupil she expects to do worse. Two or three marks a paper, in the direction of the hypothesis, is enough to manufacture a finding from nothing, and the demo at the end shows exactly how little it takes.

Blinding removes the expectation by removing the information. The twenty-four pupils' papers were marked by teachers who did not know who had the guide. The twelve-school trial made that a rule enforced by the setup: every paper went to its marker under a candidate number with the name covered, the papers were shuffled before the bundles were made up, and the list of who had the guide stayed in a sealed envelope in each headteacher's office until the last mark was in. The write-up's phrase is that blinding was "enforced by the setup, not left to discipline", which is the standard to aim for: not "I tried not to let it influence me" but "I could not have known".

Freezing the mark scheme

The mark scheme is the list of what earns credit on each question, and how much, and the moment it is written matters as much as what it says. Write it after seeing a few papers and it will drift toward the things the favoured group happens to do well, without anyone intending it: a method mark for setting out the working the way the guide teaches, say. The twelve-school trial wrote and sealed its mark scheme before a single paper was sat, and applied it unchanged to all 480.

Freezing has a companion rule for the things a mark scheme cannot anticipate. When a pupil reached a right answer by a method the scheme had not listed, the trial allowed the marker to give credit, but only for a method that was sound on the page in front of them: not for what the pupil "clearly meant", and not for an answer that would have been right had a line been copied correctly. Anything more was a mark the pupil had not earned. The rule sounds fussy and is the difference between a marker who assesses and a marker who helps.

Counting every pupil: intention to treat

Two situations come up in every study of this kind, and both invite a quiet form of cheating.

The first: a pupil is given the guide and never opens it, or is off school for a week of the fortnight and barely looks at it. Does that pupil count as "with the guide"? The trial's answer is yes, always, and the name is borrowed from clinical medicine: intention to treat. A patient who never picks up the prescription still counts in the drug group, because dropping them would leave the drug group with only the patients who were well enough to reach the pharmacy. A pupil who leaves the guide in her bag still counts in the guide group, because dropping her would leave the guide group with only the pupils who revise when told to, which is not the question. The headteacher is buying the guide for every pupil, including the ones who will never open it.

The second: some pupils sit down, panic, and hand in a near-blank paper, scoring nothing or close to it. The temptation is to set them aside as "not real attempts". Here is why that is not allowed, with numbers.

Pupils Finished the paper Mark of the ones who finished Mean counting every pupil Mean counting only finishers
Group 1 10 6 80 each 48 80
Group 2 10 10 60 each 60 60

Group 1 loses four pupils in ten and is excellent for the six who finish. Counting every pupil, Group 2 is better, 60 to 48. Counting only the pupils who finished, Group 1 is better, 80 to 60. Excluding the blank papers does not clean the data. It rewards the condition that loses the most pupils, because losing them removes its worst marks from the average. The write-up calls this a selection effect and refuses it in one sentence: excluding the blank papers "would quietly reward whichever condition produces the most of them". Every pupil whose name came out of the hat is counted, and a blank paper is a mark of zero, not a missing one.

Can someone else get the same numbers

Two words that sound alike and are not. A result is reproducible if someone with your data and your code gets your numbers. It is replicable if someone with new data and their own code gets your conclusion. The first is a matter of bookkeeping and is entirely within your control. The second is what science is for, and it is out of your hands.

The twelve-school trial's bookkeeping is three files. The anonymised marks, one row per pupil with the name replaced by a number. The mark scheme. And the analysis script, which turns the first file into every table in the write-up, so that anyone who downloads the three can regenerate every permutation p-value, every bootstrap interval and every number in the report exactly. The teacher had a colleague at another school do just that before she sent the write-up out. When a number in a report cannot be regenerated, the reader has to take the author's word for it, and the whole apparatus of this series exists so that they do not have to.

Reading a kappa

Even a blinded marker is a single marker, and the headteacher will ask whether a second pass would give the same marks. The twelve-school trial answered by having a second teacher re-mark a 10 per cent sample, 48 papers, four weeks later, still not knowing who had the guide, and reporting three numbers: 98.8 per cent of individual question marks were identical, the linearly weighted kappa was 0.973, and the intraclass correlation on each paper's total was 0.997.

Cohen's kappa is agreement corrected for luck. Two markers who each give most questions full marks will agree most of the time by chance alone, so raw agreement flatters them. Suppose they agree on 90 per cent of questions, and that from their individual habits you would expect them to agree on 60 per cent by chance. Kappa is the agreement beyond chance as a fraction of the agreement that was available beyond chance: (0.90 − 0.60) / (1 − 0.60) = 0.75. A kappa of 1 is perfect, 0 is chance, and the usual reading is that above 0.8 is excellent. Weighted kappa gives partial credit for near misses, a 3 against a 4 counting as closer than a 3 against a 0, which is right for questions marked out of several. The intraclass correlation does the same job for totals that can take any value, like a mark out of 100. At 0.973 and 0.997, the two passes were as close as two passes get, and the mean absolute difference per paper, 0.52 marks out of 100, is smaller than any effect the trial reports. That is what the numbers are for: they say that marker noise cannot be what produced the results.

Four ways a study can be wrong

Methodologists sort the ways a conclusion can fail into four kinds, and a threats to validity section walks through them. Here they are, with the questions to ask of any write-up.

Threat The question In the teacher's study
Internal Did the condition cause the gap, or did something that travelled with it? Confounds, marker expectation, pupils dropped from the count. School C's year admitted that time of day travelled with the guide. The twelve-school trial drew names by lot, covered them for the markers and counted every pupil
External Does the result hold beyond these schools, this exam and this year's pupils? Twelve schools and seven topics say more than one of each. Nothing says the guide will work at a school with a different intake, or on a differently set paper, and the write-up says so
Construct Does the outcome measure the thing the claim is about? An exam mark measures what the mark scheme lists. "Understanding maths" is a bigger phrase. The guide drills the kind of question this paper asks, so some of the gain may be practice at the paper rather than at the maths, and the write-up flags it
Statistical conclusion Were the tests, corrections and sample sizes adequate? Parts 4 to 8 of this series. The twelve-school trial's P5 is the example of a sample size that was not

A good limitations section is not an apology. It is this table, filled in without flinching, so that the reader knows which of the four the author has ruled out and which remain.

Try it: the marker who knows

The demo simulates a study in which the guide does nothing at all, and a marker who, knowing which pupils had the guide, leans by a few marks in its favour. See how small a lean it takes to produce a "finding".

Set the lean to 3 marks, a nudge too small for any marker to notice in herself, and 48 pupils a side, and press Mark 20 studies. About one in five will report a finding, from a guide that does nothing, four times the honest false-alarm rate. Push the pupils to 200 a side and it is one in two. Tick the blinding box and press again. Then notice the uncomfortable part: more pupils make the problem worse, not better, because a systematic lean is exactly the kind of small, consistent effect that large samples are good at detecting.

In Python

Kappa is one call in scikit-learn. The two small simulations after it are the two tables above: the blank-paper selection effect, and the marker who leans.

import numpy as np
from sklearn.metrics import cohen_kappa_score

# Two marking passes over the same 20 exam questions, each marked 0 to 3
first  = [3, 3, 2, 3, 0, 3, 1, 3, 3, 2, 3, 3, 0, 3, 2, 3, 3, 1, 3, 3]
second = [3, 3, 2, 3, 0, 3, 2, 3, 3, 2, 3, 3, 0, 3, 3, 3, 3, 1, 3, 3]
agree = np.mean(np.array(first) == np.array(second))
print(f"raw agreement {agree:.0%}")
print(f"Cohen's kappa {cohen_kappa_score(first, second):.3f}")
print(f"linearly weighted kappa {cohen_kappa_score(first, second, weights='linear'):.3f}")

# Why excluding the blank papers flatters the condition that produces most of them
A = [0, 0, 0, 0, 80, 80, 80, 80, 80, 80]      # 6 pupils in 10 finish the paper, and do very well
B = [60] * 10                                # every pupil finishes, every pupil is adequate
print(f"counting every pupil:    A {np.mean(A):.0f}, B {np.mean(B):.0f}")
print(f"counting only finishers: A {np.mean([x for x in A if x]):.0f}, B {np.mean(B):.0f}")

# A marker who leans 3 marks toward the guide turns a null into a finding, given enough pupils
from scipy import stats
rng = np.random.default_rng(3)
hits = 0
for _ in range(200):
    a = rng.normal(62, 15, 48) + 3          # the guide does nothing; the marker adds 3
    b = rng.normal(62, 15, 48)
    hits += stats.ttest_ind(a, b).pvalue < 0.05
print(f"a 3-mark lean with 48 pupils a side: {hits / 2:.0f}% of null studies report a finding")

which prints:

raw agreement 90%
Cohen's kappa 0.803
linearly weighted kappa 0.890
counting every pupil:    A 48, B 60
counting only finishers: A 80, B 60
a 3-mark lean with 48 pupils a side: 18% of null studies report a finding

Where this leaves us

The tests assume exchangeable labels and honest marks, and the design is what makes those true. Draw the names from a hat so that nothing else can travel with the guide, change one thing at a time, cover the names so that a marker's expectation has nowhere to go, write the mark scheme before the papers exist, count every pupil including the ones who froze on the day and the ones who never opened the guide, publish the marks, the mark scheme and the script so that every number can be regenerated, and measure your own consistency so that marker noise can be ruled out by arithmetic rather than by assurance. Then fill in the four-row table without flinching. What remains is the question of when the choices were made, which is the last part of the core series.

Next: When You Decide Matters: post-hoc, exploratory and confirmatory analysis, the freeze, preregistration and registered reports, results-blind review, and a demo in which you try to find a significant result in pure noise, then lock your choices first and try again.