student.uva.nl
Welke opleiding volg je?
Welke opleiding volg je?
Colloquiumpunten

The shopping bag problem of hypothesis testing

Colloquiumpunten

The shopping bag problem of hypothesis testing

Laatst gewijzigd op 14-10-2026 16:31
What does weighing one bag tell us when it shows not only the hypothesis but a number of other, auxiliary hypotheses?
Toon informatie voor jouw opleiding
Welke opleiding volg je?
of
Startdatum
23-10-2026 11:30
Einddatum
23-10-2026 13:00
Locatie
Imagine every empirical study as packing a shopping bag. As the project moves from hypothesis to analysis, researchers put their choices about design, measurement, data preprocessing, and statistical modelling into this bag. They then “weigh” the bag and interpret the result as evidence about the hypothesis. But what does weighing one bag tell us when it shows not only the hypothesis but a number of other, auxiliary hypotheses? Other defensible combinations could yield different conclusions - especially when we pack in assumptions without noticing. In this talk, I argue that single-design, single-analysis studies leave important uncertainties unexplored and encourage unwarranted confidence in their findings. Confirmatory analyses in these settings are premature.  I introduce Space Analysis, a framework connecting existing methods to explore alternative research choices, reduce uncertainty through validation, and integrate remaining uncertainty into our conclusions. Drawing on many-analyst studies and simulation approaches, I discuss how we can move beyond evaluating whether one analysis is defensible toward establishing how much our conclusions depend on the choices behind it.