I present a meta-analysis on the internal consistency of tests assessing domain-specific knowledge that raised a variety of applied methodological questions: 1) Can meta-analyses of reliability indices – so-called reliability generalization meta-analysis – inform us about the reliability of measurement instruments, about the internal consistency of the assessed construct, or about both? 2) Should we really abandon indices such as Cronbach`s Alpha in favor of more hip alternatives such as Omega? 3) do typical publication bias-methods such as funnel plots work for reliability indices, and if not, which alternatives exist? I present various responses and proposed solutions to these questions, relating to ongoing discussions about reflective vs. formative measurement and the abandonment of sum scores, some of which may be speculative at best and wrong at worst.