重新思考LLM导师中的脚手架：基准测试与真实部署之间的交互不匹配
Rethinking Scaffolding in LLM Tutors: The Interactional Mismatch Between Benchmarks and Real-World Deployments
AI总结：
通过分析9490个聊天记录，发现AI导师基准测试假设学生积极接受脚手架，但真实场景中学生常绕过脚手架，揭示基准测试与真实部署的交互不匹配。
摘要：
AI导师基准测试中评估的一个核心教学价值是脚手架：通过渐进步骤引导学生走向解决方案。然而，将脚手架行为嵌入聊天机器人的对齐和评估方法基于一个隐含假设：学生会接受脚手架并参与对话。为了检验这一假设是否成立，我们引入了一个围绕两个指标——聊天机器人脚手架和学生接受度——的评估流程，并将其应用于跨越AI导师基准测试和教育聊天机器人真实部署的九个数据集，共9490个聊天记录。我们的分析揭示，虽然基准测试假设一个高脚手架、高学生接受度的环境，但真实场景中的学生整体表现出较低水平的接受度——经常绕过聊天机器人的教学框架，以较低的人际成本将交互推向自己的学习目标。我们认为，绕过脚手架不一定是坏事；相反，它经常突显聊天机器人的教学框架与学生目标之间的不匹配。为了有意义地评估聊天机器人辅助的有效性，未来的基准测试必须超越学生简单接受脚手架的假设，而是评估这些聊天机器人如何应对多样化的学习环境和学生驱动的交互模式。
英文摘要：
A central pedagogical value evaluated in AI tutor benchmarks is scaffolding: guiding students through graduated steps toward a solution. Alignment and evaluation methods for embedding scaffolding behaviour into chatbots, however, rest on an implicit assumption: that students will take up the scaffolding and engage in the conversation. To examine whether this assumption holds, we introduce an evaluation pipeline around two metrics - Chatbot Scaffolding and Student Uptake - and apply them across nine datasets of 9,490 chats, spanning AI tutor benchmarks and real-world deployments of educational chatbots. Our analysis reveals that while benchmarks assume a high-scaffolding, high-student-uptake environment, students in real-world settings exhibit lower levels of uptake overall - frequently bypassing the chatbot's pedagogical framing to drive the interaction toward their own learning goals at little interpersonal cost. We argue that bypassing scaffolding is not necessarily detrimental; rather, it frequently highlights a mismatch between a chatbot's pedagogical framing and the student's learning goals. To meaningfully evaluate the effectiveness of a chatbot's assistance, future benchmarks must move beyond the assumption that students will simply take up the scaffolding, and instead evaluate how these chatbots navigate diverse learning contexts and student-driven interaction patterns.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
