Do AI language models understand what they read?

Drożdż investigated whether language models understand what they read by presenting 64 people and 16 language models with short stories in which objects move from one location to another. Can they, for example, keep track of the fact that a box is empty when the only object inside it is removed, without being explicitly told?

Looking beyond how convincing a text sounds

Her research showed that people were better at directly describing the final situation, while language models were better at identifying which continuation logically fitted a story. The models also had less difficulty with more complex stories, although in some situations they made the same mistakes as people. The research suggests that language models keep track of information about the world described in a text, but do so differently from humans. Drożdż’s findings therefore show that assessing the reliability of these models requires looking beyond how convincing their texts sound.

The other winners

The other winners of the 2026 UvA Thesis Prize were Anna Schaminée (ACTA), Dominika Karaszewska (Faculty of Medicine), Yente Tak (Amsterdam Law School), Hannah Moana Bach (Economics and Business), Kai Tjong-Ayong (Faculty of Humanities) and Rogier Schipperheijn (Faculty of Science). They will each received a prize of €1,000 for their thesis.