Imagine every empirical study as packing a shopping bag. As the project moves from hypothesis to analysis, researchers put their choices about design, measurement, data preprocessing, and statistical modelling into this bag. They then “weigh” the bag and interpret the result as evidence about the hypothesis. But what does weighing one bag tell us when it shows not only the hypothesis but a number of other, auxiliary hypotheses? Other defensible combinations could yield different conclusions - especially when we pack in assumptions without noticing. In this talk, I argue that single-design, single-analysis studies leave important uncertainties unexplored and encourage unwarranted confidence in their findings. Confirmatory analyses in these settings are premature.  I introduce Space Analysis, a framework connecting existing methods to explore alternative research choices, reduce uncertainty through validation, and integrate remaining uncertainty into our conclusions. Drawing on many-analyst studies and simulation approaches, I discuss how we can move beyond evaluating whether one analysis is defensible toward establishing how much our conclusions depend on the choices behind it.

About the speaker: Dr. Balazs Aczel (Eötvös Loránd University, Budapest)
Balázs Aczél is a Professor of Psychology and Vice-Dean for Science at Eötvös Loránd University (ELTE) in Budapest, Hungary, where he leads the Budapest Metascience Lab. His research focuses on the psychology of science, philosophy of science, and improving the transparency and credibility of scientific research.