Personalized learning path systems aim to provide content that optimizes student learning. However, there lacks research empirically evaluating different learning path recommendation systems in a live educational setting. This study compares two learning path recommendation systems utilizing Deep Knowledge Tracing (DKT), on the learning platform StudyGo. It was hypothesized that students using the new learning path system (n = 638) would show improved learning outcomes and engagement compared to students using the original learning path system (n = 590). Although the new learning path system demonstrated superior model performance, students using the new learning path did not achieve better learning outcomes or engagement over those using the original system. Further analyses revealed the new system to make more conservative predictions and generate longer learning paths, possibly leading to redundant recommendations. This research highlights that improved machine-learning model performance does not necessarily translate into improved learning outcomes for students, and that more research should be conducted evaluating various learning path systems in their influence on learning outcomes. While research exists comparing DKT models in their prediction performance, their impact on actual student learning requires further investigation to determine their practical effectiveness.