When You Decide Matters

Everything in this series so far could be done perfectly and still produce a result nobody should believe. The permutation test can be exact, the intervals honest, the correction applied, the markers blind, and the finding still worthless, if the choices that led to it were made with the result in view. This part is about timing: the difference between deciding how to analyse a study before the data exist and deciding afterwards, and the ways of proving which one you did.

The garden of forking paths

Every analysis is a sequence of small, reasonable decisions. Which outcome is the headline: the mark out of 100, the pass rate, or the mark on the hardest topic? Do the two schools get pooled or reported separately? Does the pupil who was off with flu for half the fortnight get dropped? Do you stop at 24 pupils, as planned, or add a few more because the result was "nearly there"? Each choice is defensible on its own. Each has two or three options. And at every fork, an honest researcher who has already seen the data will tend, without noticing, to take the branch that leads somewhere.

Andrew Gelman and Eric Loken called this the garden of forking paths, and their point was that it needs no dishonesty. Nobody has to try twenty tests and report the best one. It is enough to make five reasonable choices, one at a time, each slightly informed by what the data looked like, to arrive at a p-value of 0.03, meaning a gap that chance alone would produce only three times in a hundred, that would have been 0.30 down a different path. The paths that were not taken are invisible in the write-up, so the reader cannot correct for them, and Part 7 showed what uncorrected multiplicity does to the false-positive rate.

The demo makes the garden concrete. The data are pure noise: two conditions with identical true marks on every outcome. Walk the garden and see how often you can find a "finding".

Try it: walk the garden

Each checkbox is a choice you are allowed to make after looking at the data. With none ticked, the analysis is the one fixed in advance: the exam mark, both schools, every pupil, all 24. With a box ticked, you get to pick whichever option gives the smaller p-value. The button analyses a hundred datasets of pure noise and counts how many yield a significant result somewhere in the garden.

Tick all four boxes and press the button: the grid goes a third to a half red, and every red square is a report that could have been written, with a real p-value below 0.05, about a guide that does nothing. Then press Freeze every choice. The grid returns to a scatter of about five red squares. Nothing about the data changed. What changed is that the analyst no longer got to choose after looking.

The teacher's notebook from the first two years has the shape of a walk through this garden, and it is worth saying so plainly: a mark scheme she tidied between the first year and the second, a question she reworded after a trial run showed that pupils misread it, and a handful of pupils she tried a draft of the guide on before the study began, to check that its exercises fitted into a fortnight. The twelve-school trial had the same on a smaller scale, two dozen pupils at two of the schools who worked through the finished guide the term before, as a rehearsal. All of that is legitimate. It is exploratory work, and the thing that makes it legitimate is that it was done on pilot pupils whose marks never entered the analysis, and that the predictions were fixed before the first real pupil was handed a guide. Which brings us to the ways of fixing them.

Five ways to fix your choices

The options below are in order of strength, and each is the previous one plus a witness.

Conventional, or post hoc. Run the study, look at the data, decide what to report, write it up. Fast and flexible, and every analytic choice is made with the results in view, which is why a reader of a null or mixed result can always ask: would you have reported it this way if it had come out differently? School C's year is mostly this, and its write-up is honest about which analyses were post hoc, meaning decided after the fact: the topic-by-topic breakdown and the comparison of the morning and afternoon sittings, both done after the headline was known. For a study whose most likely headline is "no effect", it is the weakest position to argue from.

Exploratory then confirmatory. Split the work in two. A pilot phase in which you are allowed to look, tinker and change your mind, and a confirmatory phase in which nothing changes and the pilot data are excluded. The line between them is the freeze: the moment the hypotheses, the design, the sample size, the outcomes, the exclusion rules and the analysis code stop moving, and after which the real pupils are given their guides. The twelve-school trial did this: the pilot pupils were excluded by design, and the write-up describes the six predictions as "decided in advance". A dated notebook that records every change is good evidence of a freeze, and better evidence still if the changes stop before the first real pupil's fortnight begins.

Preregistration. The same freeze, with a third-party timestamp. You deposit the plan, hypotheses, design, sample size and its justification, outcomes, exclusion rules, analysis code, with a registry such as the Open Science Framework or AsPredicted, and the registry stamps it, so that nobody has to take your word for the date. It is not peer review and not a publication route. You still publish the write-up wherever you like afterwards and cite the registration. OSF offers several templates, including one for studies of existing data, and lets you embargo the registration for up to four years if you do not want the design public before the write-up. AsPredicted is the lighter option: nine questions, a one-page PDF, done in an afternoon.

The twelve-school trial did not do this, and says so in a sentence I would encourage anyone to copy: its predictions were "written down in advance rather than registered with anyone outside the schools". For that trial it was a fair choice, because three of its six results were strong positives that no amount of after-the-fact tuning would have been needed to produce. For the teacher's plan for next year, whose likely result is a bounded null, I think registration is mandatory rather than optional, because the equivalence bounds of Part 8 and the Holm families of Part 7 are exactly the choices the headteacher will suspect were tuned once the interval was in view.

Registered Reports. The strongest form. You submit the plan itself for peer review before collecting data, as a Stage 1 manuscript: introduction, methods, analysis plan, power analysis, and no results. If it passes, you receive in-principle acceptance: the report will be published whatever the results, provided you follow the protocol. Stage 2 adds the results and discussion, and reviewers check that you did what you said, not whether they like the answer. A growing number of journals offer the track, and Peer Community In Registered Reports runs the review independently of any one journal, with a list of journals that will publish what it recommends without further review. The costs are real. Stage 1 review adds months before the first pupil is handed a guide, and if the teacher wanted the study published she would have to choose a journal that offers the track. The benefit is precisely suited to a null-result study: a mixed or null result is guaranteed a home, and the design gets adversarial scrutiny while it can still be changed cheaply.

Results-blind review. A halfway house some venues offer: the manuscript is reviewed with the results section redacted, but after the data exist. It removes outcome bias from the reviewers, who cannot favour a positive result they have not seen, but not from the author, whose analytic choices were still made with the results in view. Less common, and worth knowing the name.

Where this is done

Medicine has required registration since 2005, when the leading medical journals agreed not to publish a clinical trial that had not been registered before its first patient was enrolled. Psychology built its registries after 2011, when a string of celebrated findings failed to repeat and the field went looking for the garden, which is where OSF and AsPredicted come from. Newer fields, computer science among them, have started preregistration tracks at their conferences in the last few years, so a teacher who lodges her plan with a registry is doing nothing exotic: she is doing what a drug trial has had to do for twenty years.

In Python

The garden, walked by a program: with one fixed, stratified path, noise reaches p < 0.05 about one time in twenty, and with twenty-four paths it does so about half the time. The last block is the cheapest possible freeze: write the plan as a file and fingerprint it, then deposit the fingerprint or the file with a registry before the first pupil is handed a guide.

import numpy as np, hashlib, json
from scipy import stats

# Walk the garden: pure noise, and four choices made after looking
rng = np.random.default_rng(20260703)
def one_dataset():
    school = np.repeat([82, 53], 12)                     # 12 pupils per school, 6 per condition
    cond = np.tile(np.repeat([0, 1], 6), 2)              # 0 = with, 1 = without: no true effect anywhere
    mark = school + rng.normal(0, 15, 24)
    algebra = 0.6 * mark + 0.4 * (school + rng.normal(0, 15, 24))   # a second outcome that moves with the first
    order = rng.permutation(24)                          # the order the pupils joined the study in
    return cond[order], school[order], mark[order], algebra[order]

def smallest_p(cond, school, mark, algebra, open_choices):
    outcomes = [mark, algebra] if open_choices else [mark]
    subgroups = [None, 82, 53] if open_choices else [None]
    stops = [24, 16] if open_choices else [24]
    drops = [False, True] if open_choices else [False]
    ps = []
    for outcome in outcomes:
        for sub in subgroups:
            for stop in stops:
                keep = (np.arange(24) < stop) & (True if sub is None else school == sub)
                centred = outcome - np.where(school == 82, 82, 53)   # stratify: remove each school's own level
                for drop in drops:
                    a, b = centred[keep & (cond == 0)], centred[keep & (cond == 1)]
                    if drop:                                    # drop the pupil who most hurts the case
                        a, b = (np.delete(a, a.argmin()), b) if a.mean() >= b.mean() else (a, np.delete(b, b.argmin()))
                    if len(a) >= 3 and len(b) >= 3:
                        ps.append(stats.ttest_ind(a, b).pvalue)
    return min(ps)

for open_choices in (False, True):
    found = sum(smallest_p(*one_dataset(), open_choices) < 0.05 for _ in range(500))
    print(f"{'24 paths' if open_choices else 'one fixed path'}: {found / 5:.0f}% of noise datasets yield p < 0.05")

# The freeze, as a file: write the plan down and fingerprint it before the data exist
plan = {"hypotheses": ["P1: with the guide vs without, exam mark, higher"], "test": "stratified permutation, 10000 shuffles",
        "correction": "Holm over the primary family", "pupils_per_cell": 6, "seed": 20260703}
text = json.dumps(plan, sort_keys=True, indent=2)
print("plan fingerprint:", hashlib.sha256(text.encode()).hexdigest()[:16], "(deposit this, or the file, with a registry)")

which prints:

one fixed path: 5% of noise datasets yield p < 0.05
24 paths: 52% of noise datasets yield p < 0.05
plan fingerprint: 9b959a1ff0b21e52 (deposit this, or the file, with a registry)

Where this leaves us

The tests of Parts 4 to 8 measure how surprising a result would be under chance. They cannot measure how surprising it would be under an analyst who chose the path, and that second surprise can be all of the first. The fix is not more statistics. It is a date: the moment after which nothing changed, and a witness to it, whether that is your own notebook, a registry's timestamp, or a reviewer's in-principle acceptance. The stronger the claim of absence, the stronger the witness needs to be.

That closes the core of the series, and Part 1's sentence can now be rewritten the way a careful study would write it. Part 1 began with: the pupils with the guide scored 12 marks higher, so it works. Here is the same claim as the twelve-school trial would make it, and every word has now been explained. Pupils with the guide scored 12.3 marks higher on the end-of-year exam than pupils without it, 95 per cent bootstrap interval 7.0 to 17.5, stratified by school and topic, Holm-corrected p = 0.0006 across six predictions written down in advance, marked by markers who did not know who had the guide against a mark scheme written before the exam, every pupil counted, the marks, the mark scheme and the analysis script published. Supported. And in the same table, three predictions that were not, reported the same way, which is how you know the first one means something.

Next: Resampling Beyond the Interval: the bootstrap as a test and how it differs from the permutation test, the paired bootstrap for conditions measured on the same topics, the bootstrap for anything you can compute, the jackknife, and where resampling lets you down. The first of five parts that widen the net.