TutorMoments: Do AI tutors know when to help and when to hold back?

The event introduces a framework that measures a previously untested dimension of AI tutors: the ability to calibrate help based on a student’s immediate needs. Unlike traditional benchmarks that reward fixed behaviors (e.g., never giving answers), TutorMoments evaluates whether a model can dynamically choose between scaffolding and rigor. This matters because over-help can short-circuit the productive struggle that deepens learning. Engineers integrating AI tutors into educational tools must now account for this trade-off, which may require prompt engineering or fine-tuning beyond generic helpfulness.

TutorMoments is built on 462 real tutoring transcripts, annotated by teachers to flag moments where tutors faced a choice between support and challenge. The framework replays these moments by handing the session to an LLM, which then interacts with a simulated student for five turns. This replay-based approach isolates the model’s decision-making from other variables, but it also introduces limitations: the simulated student may not fully capture real student behavior, and the five-turn window may not reflect longer-term pedagogical strategies. Adopting this framework requires access to similar annotated data, which may not exist for all subjects or age groups.

The framework reveals that models prompted with generic instructions (e.g., 'tutor well') tend to over-help, while explicitly framing the trade-off improves performance. However, even with this guidance, models still fall short of human tutors’ consistency. This suggests that current LLMs lack the contextual awareness to adapt to individual students, a gap that may persist until models can better infer student understanding from dialogue. For engineers, this means that off-the-shelf LLMs may not be sufficient for tutoring applications without additional layers of decision logic or human oversight.

The release of the dataset, code, and model replays enables reproducibility and comparison across models. This is valuable for researchers and developers, but it also highlights a broader challenge: evaluating AI tutors requires more than technical metrics. The framework’s reliance on teacher annotations underscores the importance of domain expertise in designing AI for education. Engineers must collaborate with educators to define what constitutes 'good' tutoring in their specific context, as the framework’s current focus on math tutoring for grades 2-7 may not generalize to other subjects or age groups.