student.uva.nl
What is your study programme?
What is your study programme?
Colloquium credits

The shopping bag problem of hypothesis testing

Colloquium credits

The shopping bag problem of hypothesis testing

Last modified on 14-10-2026 16:31
What does weighing one bag tell us when it shows not only the hypothesis but a number of other, auxiliary hypotheses?
Show information for your study programme
What is your study programme?
or
Start date
23-10-2026 11:30
End date
23-10-2026 13:00
Location
Imagine every empirical study as packing a shopping bag. As the project moves from hypothesis to analysis, researchers put their choices about design, measurement, data preprocessing, and statistical modelling into this bag. They then “weigh” the bag and interpret the result as evidence about the hypothesis. But what does weighing one bag tell us when it shows not only the hypothesis but a number of other, auxiliary hypotheses? Other defensible combinations could yield different conclusions - especially when we pack in assumptions without noticing. In this talk, I argue that single-design, single-analysis studies leave important uncertainties unexplored and encourage unwarranted confidence in their findings. Confirmatory analyses in these settings are premature.  I introduce Space Analysis, a framework connecting existing methods to explore alternative research choices, reduce uncertainty through validation, and integrate remaining uncertainty into our conclusions. Drawing on many-analyst studies and simulation approaches, I discuss how we can move beyond evaluating whether one analysis is defensible toward establishing how much our conclusions depend on the choices behind it.