Design

I think this is the most important step. You should know what you want.

My first priority was to achieve the same convenience I have had using Stata’s ritest, which required a simple public API. My second priority was flexibility, which was again a design concept taken from Stata’s ritest. This is what would end up being the ‘linear’ and ‘generic’ path. Finally, my last priority was speed, which is at odds with the second priority. Conveniently, the separation between ‘linear’ and ‘generic’ path provided a natural solution. I would guarantee speed on the linear path.

Tools

This section is a description of the tools that I used. This is not a tutorial on how to install or use these tools; I’m not the right person to do so. The section simply sets the stage for the next sections, in which I precisely describe my development workflow. You can safely skip this section if you are familiar with DevOps.

Workflow

Now that I’ve described the tools, I can tell you about my (solo) development workflow. It represents the workflow for a new feature or change, for a very minor change or at earlier stages, you can just skip the new branch.

By the way, I wrote ritest during 2025 with support from Open AI’s Codex. Things may have changed with the new models, but at the time, Codex needed close oversight and made lots of mistakes. Not to say it was not a great tool; this package would be worse (or even a never completed project) without LLMs. Like all the tools mentioned in this post, it is just another way to make the job easier.

flowchart TD
  A[Create branch] --> B[Edit code]
  B --> C[Run pre-commit]
  C --> D{Hooks changed files?}
  D -- yes --> E[git add -A]
  E --> F[Commit]
  D -- no --> F[Commit]
  F --> G[Run tests: pytest]
  G --> H{Tests pass?}
  H -- no --> B
  H -- yes --> I[Merge into main/master]
  I --> J[Push]
  J --> K{CI green?}
  K -- no --> B
  K -- yes --> L[Done]

The workflow corresponds to:

git checkout -b feat/x
# edit code
pre-commit run --all-files
pytest
git add -A
git commit -m "message"
git checkout main
git merge feat/x
git push

Packaging and release on PyPI

Once you are ready to ship, you need a pyproject.toml file. This is where you tell Python tools what your project is and how it should be built. It declares project metadata (name, version, description, URLs), minimum Python version, and runtime dependencies.

To release on PyPI, you can follow these steps:

• build wheel + sdist locally
• upload to TestPyPI with twine (this is ‘rehearsal’)
• install into a clean environment and actually use it
• upload the same artefacts to PyPI
• bump version + tag release
• Done!

Documentation

The second lesson I took from my time as a CS student is that you must write proper documentation. So I also did that. And it takes much longer than what I thought.

Beyond proper in-code documentation, I think, at the very least, you need a README file for your GitHub repository. It describes what the package does, how to install it, and how to use it. Then, you need a CHANGELOG file to keep track of changes. For me, the real work was to write a comprehensive documentation site, built with Quarto and deployed via GitHub Pages. This is where I show basic and advanced use, examples, technical notes, and the reference API.

One last word

I hope that I’ve clearly conveyed that this is not a “how it is done” post. I just did it for the first time.I am sharing this post for selfish reasons. I would like to hear from people who have done this many times. Am I missing something that would make the process easier, faster, or more robust? Please reach out if you have something to say.

And if you are thinking about releasing your first package, you may find this post useful, just keep in mind that this is in no way an authoritative guide.

-values

Randomisation inference (RI) starts from a statistic : a difference in means, a regression coefficient, a median difference, or anything else you care about. The design (the assignment mechanism) induces a randomisation distribution for that statistic under a null hypothesis.

Under a sharp null, the randomisation -value is a tail probability under the assignment mechanism:

If you can enumerate every valid assignment, you can compute exactly, in the design-based sense. In most real problems there are too many valid assignments and it is not worth it to go through all of them, so you sample valid reassignments.

Let be the number of assignments in which the statistic is more extreme than the observed statistic: A common Monte Carlo estimator is¹

At this point it is no longer correct to treat the reported -value as a fixed number. It has Monte Carlo error because is finite. In contrast, in an OLS regression, the -value is a deterministic function of the data, given the modelling assumptions. So there is no Monte Carlo uncertainty.

It is worth pointing out that -values, both in RI and the regression context, only make sense in the context of a hypothesis test. A -value is not a generic measure of ‘signal strength’; it is defined relative to a null hypothesis and a reference distribution for the statistic. In RI, both are explicit: the null is sharp, and the reference distribution comes from the assignment mechanism. In the regression context, the null is typically weak and the reference distribution is introduced analytically (often via asymptotic arguments).

Confidence intervals

We now have our Randomisation inference (RI) -value, which is an estimate with Monte Carlo error. Then, it is natural to represent this error with a confidence interval (CI) for the -value itself.

Conditional on the observed data and the null, each reassignment either lands in the tail or it does not. That makes behave like a binomial count: Then, you can build a interval for from this binomial model (there are several standard choices).

The interpretation is narrow but clean. The RI confidence interval of the -value:

• quantifies uncertainty from simulation, not from (theoretically) drawing a new dataset,
• shrinks as grows, and
• tells you when a ‘statistical significance’ call is robust versus when you are basically flipping a coin near a threshold.

Now let’s get back to the regression setting, say, for an A/B test. How do we build a confidence interval?

To be concrete, define the finite-sample average treatment effect

If you estimate the effect as a treated–control difference in means, or as the coefficient on a treatment indicator in an OLS regression with an intercept, you are targeting on the outcome scale. A standard regression confidence interval takes the form with details depending on how you estimate the variance.

The key thing is what that CI is trying to cover. In the usual regression presentation, the motivation is based on repeated sampling (often asymptotic): if we reran ‘the relevant randomness’ many times, the interval would cover the target parameter with frequency . The ‘not significant if the CI includes 0’ common saying is shorthand for not rejecting a weak null like under that sampling-based uncertainty and approximation. That is an uncertainty statement about an average effect.

Let’s take a moment for the distinction to sink in. The confidence interval in a typical regression table reflects uncertainty around the coefficient, while the (default) confidence interval in randomisation inference reflects uncertainty around the -value. They are not at all comparable.

Confidence set

It looks like we are missing a piece in randomisation inference. Is there no confidence interval for the coefficient? Well… sort of. At least in spirit. But we need to do much more work, both to build it and to interpret it.

A randomisation test needs a null that lets you impute missing potential outcomes. The canonical ‘no effect’ sharp null is That is stronger than ‘the average effect is 0’—the ’weak‘ null. It says nobody is affected.

To get an interval for an effect size, RI typically inverts a family of sharp nulls indexed by a candidate constant additive effect:

For each you compute a randomisation -value . Then you invert the tests:

In other words: A ‘confidence interval’ for the coefficient under randomisation inference is the set of constant additive effects that you do not reject under design-based uncertainty.

That is what it means mechanically, and it is also the safest way to interpret it.

Everything else—especially ‘is it an ATE interval?’—depends on whether the constant-effect assumption is a defensible approximation for your application.

Note that the ‘confidence interval’ in randomisation inference is literally defined as a set, which explains that in randomisation inference we call the boundaries of that set the ‘confidence bounds’ instead of ‘confidence interval’. In addition, we call the whole set a ‘confidence band’. This keeps the whole inversion visible: plot the curve and draw the horizontal line at . The confidence set is where the curve sits above the line. I like bands because they answer questions the bounds cannot, such as

• do the endpoints come from a sharp crossing or a curve that barely grazes ?
• is the acceptable region a single chunk, or does it fragment?
• if is computed by Monte Carlo, is the crossing stable once you acknowledge simulation noise?

Hypothetical example

Think of a product A/B test at TikTok. A new onboarding flow is rolled out to a random subset of users, and you want an assignment-based uncertainty statement for the treatment effect.

• unit: user
• treatment indicator: treat (1 = new onboarding, 0 = old)
• primary outcome: activated_7d (1 = activated within 7 days)
• pre-treatment covariates (optional, for precision): pre_usage (numeric), device_ios (0/1), region_eu (0/1)

If the experiment used blocked randomisation, you also have a strata variable:

• strata / blocks: strata_id (e.g., country-by-device buckets used in the randomisation)

If the experiment randomised at a higher level (say, by creator cohort or by market), you may also have:

• cluster: cluster_id

The statistic I will use is the treatment coefficient from

Stata

You can install the command ritest from the SSC archive:

ssc install ritest

Below, ritest permutes the assignment variable treat, re-runs the estimation command each time, and collects the statistic of interest. The typical pattern is:

• write the model as you usually would
• tell ritest which coefficient/statistic to track
• optionally enforce strata / clusters to mirror the design

ritest treat _b[treat], reps(5000) seed(123) ///
    strata(strata_id) cluster(cluster_id) nodots : ///
    regress activated_7d treat pre_usage device_ios region_eu, vce(cluster cluster_id)

Notes:

• The cluster() option in ritest refers to the randomisation unit if assignment is clustered; the vce(cluster ...) inside regress is a modelling choice for conventional standard errors, not the RI itself. The latter does not have implications for randomisation inference.
• If you did not randomise within strata, drop strata(strata_id). If you did not cluster assignment, drop cluster(cluster_id).

R

You can install the package ritest from GitHub

remotes::install_github("grantmcdermott/ritest")

In R, the pattern is usually:

1. fit a model object (often lm() or fixest::feols())
2. pass the fitted object to ritest(), specifying the resampling variable and (optionally) strata/cluster structure

library(ritest)

fit <- lm(
  activated_7d ~ treat + pre_usage + device_ios + region_eu,
  data = df
)

ri <- ritest(
  fit,
  resampvar = "treat",
  reps = 5000,
  strata = "strata_id",
  cluster = "cluster_id",
  seed = 123
)

ri

This mirrors the Stata workflow: the statistic is defined through a fitted model object, and RI is performed by permuting the assignment in a way that respects the experimental design.

pip install ritest-python

from ritest import ritest

res = ritest(
    df=df,
    permute_var="treat",
    formula="activated_7d ~ treat + pre_usage + device_ios + region_eu",
    stat="treat",
    strata="strata_id",
    cluster="cluster_id",
    reps=5000,
    seed=123,
)

print(res.summary())

Notes

I think that the sequential development of these implementations has made it very convenient for a user to move around the three different languages. They all support very similar features and usage patterns, including ways to define generic statistics.

Despite these functional similarities, there are significant implementation differences. Some are necessary differences imposed by the software environment, and others reflect deliberate design and implementation choices. Because randomisation inference is, at its core, repeated re-estimation of the statistic, these differences can have a significant impact on performance. On a shared real dataset example, my documentation reports approximate wall-clock times of about 220 seconds (Stata), 16.45 seconds (R), and 7.28 seconds (Python) for 5,000 permutations with a fixed-effects style specification and clustered and stratified design constraints. That said, performance is contingent on the specific computations for a given statistic.

Which one should you use? Most likely, whatever you were planning to use. For most applications, it would not make sense to choose your software based on the performance of the available ritest. Stata is certainly not well known for its speed, but it remains my favourite language for data analysis.

Theory

Professional leagues like the WNBA are best understood as a legally constrained joint venture that functions as a monopoly platform in the market for top-tier basketball while also operating as a dominant buyer of elite basketball labour. Because a league season is a joint product—teams are rivals on the court but complements in production—many core choices are centralised: rules of play, scheduling, media-rights packaging, revenue sharing, and restrictions on labour mobility. As a result, wages do not emerge from decentralised market clearing. They are set inside an administered internal labour market shaped by league governance (draft assignment, rookie scale, restricted free agency, salary cap and maximum contracts), which both manages competitive balance (a demand-side feature of the product) and compresses the set of feasible contracts relative to an open auction.

Player pay is therefore the equilibrium outcome of collective bargaining in a bilateral-monopoly environment: the league/teams act as a coordinated buyer through shared rules, and the union coordinates labour supply. The bargaining set is pinned down by basketball-related revenues (especially national media and sponsorship income) and by the league’s commitment to accounting definitions and cap architecture; the division of surplus is governed by threat points and outside options. Owners’ leverage comes from capital depth and the ability to withstand short-run losses, while players’ leverage comes from the star elasticity of demand, reputational and broadcast losses from a stoppage, and whatever outside earnings or alternative playing opportunities raise reservation pay. Negotiations are thus about the rent split and its incidence across player types—how the cap, maximums, minimums, and exceptions allocate revenue growth between superstars, mid-tier players, and marginal roster players—rather than ‘price discovery’ in any competitive sense.

Practice

This is much easier than the theory. What do we know in practice? I’ll cover three areas: the players’ experience, the league’s reported losses, and the investors’ bullish attitude.

Current negotiation

The WNBA’s last proposal, as reported, looks generous at first glance because it roughly offers a four-fold increase in salaries across the board, with an average salary around half a million dollars. The players rejected the proposal. Instead, they demand a ‘fair’ share of the ‘cake’, as well as a clearly defined cake.

For context, in the NBA, there is a clearly defined cake: basketball-related income. This cake has been shared in a roughly 50/50 split since 1983, when the league revenues were around $118 M. Based on the sell of the Kansas City Kings to a group from Sacramento in 1983 for $10.5 millions as an anchor price, we can multiply that price by the 23 teams in the league at the time, to obtain a total of $241 millions, which correspond to about $781 millions in 2025. Even assuming that the anchor price was below average, this back-of-the envelope calculation suggests that the NBA was valued under a billion dollars when the players negotiated a roughly 50/50 split. Again, the WNBA today is estimated to be about $3.5 billion.

The players’ demands focus on revenue sharing. They want a similar definition of the cake, as well as a share of it. In particular, they reportedly seek a 30/70 split. Still below the NBA’s 50/50 split.

The players have also reported discontent with the ‘disrespectful tone’ of the negotiations, in which the league has repeatedly argued that the players do not understand the situation, repeatedly returning to the operating losses in which they seem to want the negotiation to be anchored.

Unfortunately, the league’s narrative of confused players seems to resonate with the public. As I have argued in this piece, the theory behind the price determination in this case is rather complex, but there is credible empirical evidence about the investor’s bullish attitude towards the WNBA. It is then puzzling (or is it?) that people argue that the WNBA players don’t understand a situation that directly and greatly affects them. Never mind that most players have completed 4-year degrees.

A proposal

There has always been value in the WNBA. This was recognised in the mid-1990s, when the NBA launched the WNBA as a “strategic extension of its basketball platform”. My interpretation is that the motivation was not “to help women’s basketball”, but to control the product, timing, and branding of women’s pro basketball in the US. The value remains today, greater than ever, as shown by the recent growth in market value.

So… I think that the players are asking for too little. I say they should ask for almost the same deal the NBA gets. I don’t see a strong economic reason for large differences in draft eligibility, revenue-sharing split, salary cap strictness, salary structure, and player movement or retention tools.³

Along with the “bold” demand of equality, the players may benefit from steering away from the league’s focus on the operating loss. Instead, bring the focus to the owners and their capital gains. More specifically, bring the NBA to the spotlight. After all, the NBA owns about 42% of the WNBA (outside team owners own ~42% and outside investors the remaining ~16%). When talking about ‘owners’ in abstract, there is no reputation at stake. The NBA’s reputation should be at stake.

Furthermore, there are credible alternatives to the WNBA, including the Athletes Unlimited Pro Basketball league, Unrivaled, and Project B, as well as established leagues outside the US. These alternatives provide real outside options and therefore meaningful leverage in negotiations. The same strategic logic that led the NBA to launch the WNBA in the 1990s remains relevant today: controlling the development of a growing adjacent market can be more valuable than maximising short-term operating margins. From the NBA’s perspective, the risk is not that a challenger is imminent, but that suppressing player compensation in a rapidly growing women’s game increases the probability that future growth occurs outside its institutional umbrella. In that sense, conceding meaningful equality to WNBA players is less a concession than a hedge against long-term competitive and reputational risk.

Finally, there is a channel of value creation that is largely absent from the current framing: demand expansion. For most of the league’s history, playing in the WNBA has been closer to a “for the love of the game” proposition than a financial one, at least for the median player. In my view, that matters because former players—especially those who played the sport growing up—are likely to be among the most durable and loyal consumers of basketball over a lifetime, even if this effect is gradual and cohort-driven rather than immediate.

In a league whose domestic fan base has historically skewed male, the WNBA represents a structurally underexploited opportunity to broaden basketball consumption by bringing in new fans rather than reallocating existing ones. This is not simply about gender composition—differences there appear modest—but also about age and entry points into fandom, where WNBA audiences tend to skew younger, which is particularly valuable for long-run demand. This perspective is especially relevant for the NBA today, as much of its recent growth appears to be driven by international expansion rather than domestic market deepening.

From that standpoint, the WNBA is not only an asset whose value lies in media rights and franchise appreciation, but also a long-horizon investment in expanding the overall basketball audience in the US and beyond. Underinvesting in player compensation risks slowing that process. Viewed this way, improving the economic terms for WNBA players is not primarily about fairness or bargaining power; it is a strategic investment in future demand.

Randomisation inference

When an experiment is randomised, there are two different stories you can tell about uncertainty.

One story is the ‘sampling’ story. You imagine your dataset as one draw from a larger population, and you ask what would happen if you could repeat the data-collection process. That is the story behind most textbook standard errors and t-tests.

The other story is the ‘assignment’ story. You hold the outcomes fixed and ask what would have happened under different random assignments of the same treatment. That is the story behind randomisation inference.

Operationally, randomisation inference is simple:

1. pick a statistic that measures the effect you care about
2. compute it on the observed assignment
3. recompute it under many alternative assignments that respect the experimental design
4. compare the observed statistic to its randomisation distribution

That’s it. The hard part, in practice, is doing it in a way that is fast enough to use, and strict enough about the design to be trustworthy.

Features

ritest supports two ways of defining the test statistic. In the most common case, the statistic is a coefficient from a linear model, specified through a regression formula. When that is not appropriate, you can instead provide a custom Python function that maps the data to a single scalar statistic.

In both cases, permutations can be constrained to respect the experimental design, including stratified randomisation, clustered assignment, and optional weighting on the linear path.

By default, ritest makes the Monte Carlo uncertainty in the p-value explicit when permutations are sampled rather than enumerated (which is almost always true). In that case, the p-value itself is an estimate, and the output includes a confidence interval for that estimate. On the linear path, the package also reports coefficient bounds (or a confidence interval) by default.

The package can be installed from PyPI:

pip install ritest-python

Example

Here is a realistic pattern from product work. Imagine a rollout where users are randomised to see a new onboarding flow. The outcome is whether the user activates within 7 days. You also have pre-treatment covariates that help with precision (previous activity, device type, country). The effect you want is the coefficient on treat.

import pandas as pd
from ritest import ritest

# Example column meanings:
# - activated_7d: 0/1 (activated within 7 days)
# - treat: 0/1 (assigned to new onboarding)
# - pre_usage: numeric (pre-treatment engagement)
# - device_ios: 0/1 (pre-built dummy; you can build dummies upstream)
# - region_eu: 0/1 (pre-built dummy)
# - strata_id: str/int (block or bucket used in the randomisation)

res = ritest(
    df=df,
    permute_var="treat",
    formula="activated_7d ~ treat + pre_usage + device_ios + region_eu",
    stat="treat",
    strata="strata_id",
    reps=5000,
    alpha=0.05,
    seed=23,
)

print(res.summary())

This is the workflow I wanted: I can express the estimand as a familiar regression coefficient, and I can get assignment-based uncertainty without pretending the only randomness in the problem is sampling noise.

Now imagine that the adoption question is not your bottleneck. Your bottleneck is latency: you care about the median time-to-value, which is skewed and full of long tails. You still have a randomised assignment, but you do not want to force the problem into a linear model.

That is what the generic path is for.

from ritest import ritest

def median_diff(d):
    treated = d.loc[d["treat"] == 1, "time_to_value_hours"].median()
    control = d.loc[d["treat"] == 0, "time_to_value_hours"].median()
    return treated - control

res = ritest(
    df=df,
    permute_var="treat",
    stat_fn=median_diff,
    reps=5000,
    alpha=0.05,
    seed=23,
)

print(res.pvalue)

The point is not that medians are “better” than conditional means. The point is that a real workflow often has both kinds of questions, and the underlying source of uncertainty (the assignment) is the same.

Conclusion

I built this package because I needed it. The project grew well beyond my original plan as I tried to emulate, in Python, the same sense of convenience I had relied on when doing randomisation inference in Stata. I’m happy with the result, and I hope others find it useful. Since this is my first time releasing a package on PyPI, I genuinely want to hear what people think.

Finally, I want to encourage data scientists, data analysts, and researchers who are not familiar with randomisation inference to take a closer look. Randomisation inference can be appropriate whenever assignment is controlled and known. This is a common setting in many contexts: A/B testing in product and platform experiments, randomised controlled trials in economics and political science, greenhouse and field experiments in agricultural science, and laboratory or clinical studies in life sciences. If the main source of uncertainty in your problem comes from the design itself, randomisation inference may be right for you.

Inference is a thought experiment

Inference is always a thought experiment. In our example, we get a point estimate for that experiment, for those users, over that window. What if we had another experiment? What if we had other users? What if we had another window? Unfortunately, that we cannot see. And so we rely on thought experiments.

Confidence intervals and -values answer those ‘what if’ questions within a given thought experiment: What would we see if the world replayed repeatedly, in some relevant sense? That replay is not a minor detail. It is the definition of what your uncertainty statement means. And in most settings there are two replay modes that make immediate sense.

Mode 1: replay the users. Imagine Spotify could re-run the same experiment many times, but each time the platform happens to see a different slice of users: different people are active, eligible, reachable, or simply online during your windows. You run the same A/B each time, and your estimate moves around because the people changed. This is sampling-based uncertainty.

That story corresponds to the classical statistical inference we typically encounter in textbooks and beyond: -values motivated via repeated sampling. The key idea is that your effect (point estimate) could have been different had your sample of users been different. That is the uncertainty you are trying to estimate.

Mode 2: replay the assignment. Now hold the users fixed. Imagine Spotify could take the same users, same window, and same (potential) outcomes. The only thing you replay is the randomisation: who got the ‘Audiobooks’ shelf and who didn’t, respecting whatever rules you originally used (such as equal split, stratification, blocked randomisation, and so on).

That story is the realm of randomisation inference. It is also the clean way to interpret permutation tests in an experiment: you are not permuting “because it is non-parametric”; you are generating the distribution of your statistic under the assignment mechanism you actually used.

Quantifying uncertainty

I find it useful to think about inference in three layers. The first layer is the estimator (or statistic): it produces an estimate of a target effect . You need something to make an inference about. Furthermore, random assignment lends causal credibility to the interpretation of the estimate. The second layer relates to the scope of the inference. Are we trying to make inferences about the broader population we want to generalise to? (replay mode 1) Or are we trying to make inferences about who happened to see the ‘Audiobooks’ shelf due to the particular realisation of the randomisation process? (replay mode 2) Or maybe both? The third layer refers to the quantification of the uncertainty within the scope of the inference. It is here that we can find the many methods that take the first two layers and turn them into -values and confidence intervals.

For example, in our hypothetical experiment, the first layer is the estimator. We compute the treatment effect estimate , for instance as the OLS coefficient on the treatment indicator, which (with an intercept) is algebraically equal to the treated–control difference in means,

Suppose that, in the second layer, we adopt a sampling-based replay story: the estimate would have been different had the experiment observed a different random sample of users.

In the third layer, we can quantify that uncertainty in several ways; with its interpretation being contingent on the sampling-based story.

A standard error is an absolute measure of dispersion. It estimates the variability of the estimator across repeated samples,

A confidence interval converts the same idea into an absolute uncertainty range for the estimand (the target effect). Under a Normal approximation, a confidence interval for is where denotes the corresponding quantile of the standard Normal distribution (or the appropriate quantile in finite samples).

A -value is different in nature: it is defined only relative to a hypothesis. If we wish to assess compatibility with a specific reference value (often ), we form the standardised statistic

Under the sampling-based assumptions and the chosen reference distribution, the -value is that is, the probability—computed under the null hypothesis—that a re-sampled experiment would produce a standardised statistic at least as extreme as the one observed.¹

So far, this may look like ‘methods’: -tests, confidence intervals, p-values. But the three layers are the point. None of these outputs make sense in isolation. The meaning comes from (i) the estimator, (ii) the replay story, and only then (iii) the calculator used to turn the story into numbers.

Uncertainty calculators

Now that the estimator and the replay story are fixed, the remaining question is how to compute uncertainty within that scope. This is where most methods people recognise live. They are mostly different ways of approximating the same object: the distribution of under the chosen replay mode.

There are two broad ways to get that distribution.

Route A: estimate variability, then approximate a reference distribution. This is the standard error route. You compute , form a standardised statistic, and then map it to a -value (or CI) using a reference distribution (Normal or in simple cases). This family includes the classic -test and its close relatives (Wald tests, tests, tests), all of which share the same structure:

Within this route, you still have choices about the variability estimate. In regression output, for instance, a model-based OLS standard error is tied to a particular noise model, while a robust (sandwich) standard error is designed to be less dependent on that model. The estimator can be identical, while the attached uncertainty calculation changes because the calculator changed. That is why Freedman’s (2008) warning matters even in experiments: randomisation can justify the causal meaning of , while leaving room for disagreement (or mistakes) about the standard error attached to it.

Route B: build a reference distribution directly by replaying the world. This is the resampling or re-randomisation route. Instead of estimating an and leaning on a Normal or approximation, you generate many ‘replays’ and recompute the statistic each time. The output is an empirical distribution of , from which you can read off uncertainty summaries.

Two big families sit here:

• Bootstrap and jackknife (sampling replay): you replay which users you observed by resampling units (bootstrap) or systematically leaving them out (jackknife). You can then compute as the standard deviation of the replicated estimates, or compute confidence intervals from quantiles of the empirical distribution. A -value is also possible, but it requires an explicit hypothesis construction, just like before.

• Randomisation inference (assignment replay): you replay who was treated by re-running the randomisation procedure many times, respecting the original design. Under a sharp null, this directly gives a reference distribution for your statistic under the assignment mechanism.² A -value can be computed with the most literal tail probability: where is the number of simulated reassignments and is the statistic under reassignment . Notice what is missing: there is no required step of ‘estimate an and assume Normality’. The design supplies the reference distribution.³

Practical implications

You may have felt something off up to this point. In our Spotify example, the effect estimate is justified by random assignment (design logic), while a lot of standard inference is presented through a sampling lens. Furthermore, in plain A/B tests, you often find that robust -tests, bootstrap uncertainty, and randomisation-based checks all tell the same story.

This is not because the layers collapse into one. It is because the situation is unusually ‘friendly’:

• The estimator is simple. A difference in means is a stable object.
• Sample sizes are large. Many distributions become well-behaved once you have enough users, and many reasonable standardisations start to look similar.
• Different variance calculators converge. In the binary-treatment case, several common standard-error formulas are built from the same ingredients (treated and control variability and group sizes), so their numerical differences can get washed out.

If all you need is a quick answer to ‘did the shelf move audiobook listening?’, this is why your preferred software’s default often feels like it ‘just works’.

But the friendly zone is not guaranteed. The moment the design stops being “randomise users 50/50”, the replay world changes, and the calculator has to match it.

For example, if you randomised within strata (country, device, prior engagement), then ‘replay the assignment’ means reshuffling within strata. A calculator that ignores this is quantifying uncertainty for a world that never could have happened. Alternatively, if assignment happens at a higher level (households, classrooms, markets), the effective sample size is the number of clusters, not the number of users. Many default approximations become fragile when there are few clusters.

This is the practical take of the three layers: inference is not a button you press after you get . It is the combination of (i) what you estimated, (ii) what you think could have happened, and (iii) how you chose to quantify that.

Reading list

The references below have been helpful to me; they are not in any way meant as a comprehensive survey.

⭐ Abadie, A., Athey, S., Imbens, G., and Wooldridge, J. 2020. “Sampling-Based versus Design-Based Uncertainty in Regression Analysis.” Econometrica. link

Athey, S., & Imbens, G. W. (2017). The econometrics of randomized experiments. In Handbook of economic field experiments. North-Holland. link

Freedman, David A. 2008. “On Regression Adjustments to Experimental Data.” Advances in Applied Mathematics. link

Imbens, G. W., & Rubin, D. B. (2015). Causal inference in statistics, social, and biomedical sciences. Cambridge university press. link

Spotify. 2025-03-13. “How Spotify Is Driving Growth, Discovery, and Innovation in the Audiobook Market.” Spotify Newsroom. link