The increasing integration of artificial intelligence (AI) into online learning platforms has created new opportunities to provide scalable and personalized academic support to students. Despite the growing adoption of AI-based tutoring systems, empirical evidence regarding their impact on student learning outcomes remains limited. In particular, little is known about how AI tutors affect learning when students interact with them voluntarily in natural platform usage contexts. This study examined the effect of an AI-based tutor on student learning outcomes within the online learning platform StudyGo. 

Using platform log data, learning outcomes were operationalized as students’ correctness on topic-related practice exercises. To account for self-selection bias in students’ decisions to use the AI tutor, propensity score matching was applied to construct comparable groups of tutor and non-tutor user-topic attempts. In addition, a large language model (LLM)-assisted procedure was developed and validated to identify tutor-delivered practice questions within tutor-student conversations, allowing learning outcomes to be measured even when students practiced inside the tutor rather than through standard platform exercises.  

The final matched sample consisted of 484 tutor attempts and 484 non-tutor attempts. Results from a linear mixed-effects model indicated no statistically significant effect of AI tutor usage on topic correctness. Thus, AI tutor usage was not associated with higher correctness after accounting for clustering within users and topics. These findings suggest that access to an AI tutor alone may not be sufficient to improve immediate exercise performance under realistic platform usage conditions. By leveraging large-scale observational data and causal inference techniques, this study contributes to the growing literature on AI-supported learning. Furthermore, it provides a methodological blueprint for evaluating AI tutoring systems in online learning platforms where randomized experiments are difficult to implement.