TutorMoments: Do AI tutors know when to help and when to hold back?

Details

The Allen Institute for AI (Ai2) has released TutorMoments, a replay-based evaluation framework that tests whether LLMs can make the central pedagogical trade-off in tutoring: when to provide support versus when to push a student to reason independently. The benchmark is grounded in real one-on-one math tutoring sessions collected from a US tutoring program, with experienced math teachers flagging specific moments where this decision had to be made. Each flagged transcript is handed to a language model playing the tutor role, with a separate LLM playing the student. The findings are consistent: LLMs over-help, providing explanations and answers rather than the questions or partial scaffolding that effective human tutors use to promote productive struggle. Immediately giving students the answer cuts short exactly the kind of effortful problem-solving that learning research shows is most effective. Adding explicit instructions about the help-vs-hold-back trade-off in the tutor prompt improves performance, but does not eliminate the gap. Models also differ substantially in how reliably they make the right call across sessions. Ai2 has released the de-identified tutoring transcript dataset, the replay pipeline code, and recorded model tutor sessions for reproducibility — giving researchers and edtech developers a concrete benchmark for a previously hard-to-measure pedagogical skill.