Depression is a leading cause of disability worldwide. A core methodological assumption underlying depression treatment research is that depression scales are interchangeable, yet scales differ substantially in which symptoms they measure, how much weight they assign to each symptom, and what theoretical assumptions they embed. 

Using an existing individual participant data database of 21 randomised controlled trials of internet-based cognitive behavioural therapy for depression, this study examines agreement between four pairs of depression scales, BDI-II/PHQ-9, CES-D/PHQ-9, IDS/PHQ-9, and BDI-II/MADRS, using an inter-rater reliability framework. Agreement was assessed via the Intraclass Correlation Coefficient on standardised treatment-induced change scores, with Bland-Altman plots used to examine the limits of agreement and potential proportional bias. 

Moderate agreement was observed across all four pairs, with credible intervals spanning from poor to moderate agreement for three pairs and poor to good for BDI-II/MADRS, such that poor agreement could not be ruled out for any pair. Bland-Altman plots further indicated that two scales can produce substantially different estimates of change for the same individual, with proportional bias observed for three of the four pairs. 

These findings indicate that widely used depression scales are not interchangeable, and that study outcomes depend, at least in part, on which scale is used. Scale non-interchangeability should be treated as a factor to be explicitly accounted for in future research.