One of the most human-like traits about modern large language models (LLMs) is a phenomenon called in-context learning (ICL), where a model spontaneously learns a new task from just a few examples. To study the mechanisms that underlie ICL, we focus on the discrimination-shift task (DST), a paradigm from comparative psychology with a rich history of studying and comparing learning mechanisms across many different species. Applying this paradigm to LLMs, we observe large, qualitative differences across LLMs. 

Only for the strongest performing models, we find moderate to strong evidence that LLMs, like humans, learn the task by testing hypotheses based on abstract rules. For poorer performing models, we observe large inconsistencies across replications, indicative of brittleness likely connected to random ordering of trials. Overall, our findings support the hypothesis that ICL, for successful models, is facilitated by abstractions which in turn unlock strong generalization and adaptability.