如何做到？
"How Do I ...?": Procedural Questions Predominate Student-LLM Chatbot Conversations
AI总结：
本文研究了学生与LLM聊天机器人对话中程序性问题的主导地位，发现其在总结性评估中更为突出，但现有分类方案存在局限性。
摘要：
通过基于大型语言模型（LLM）构建的教育聊天机器人提供支架具有潜在的风险和益处，这些方面仍是一个开放的研究领域。当学生遇到障碍时，他们会通过提出障碍驱动的问题来寻求帮助。在与LLM聊天机器人的互动中，此类问题塑造了用户提示并驱动聊天机器人响应的教育有效性。本文聚焦于两种不同学习情境下的学生问题：形成性自学和总结性评估作业。我们分析了来自两种学习情境的6113条信息，使用11种不同的LLM和三位人类评估者，根据四种现有方案对学生成问题进行分类。在使用LLM作为评估者可行性方面，结果表明中等至良好的评估者一致性，比人类评估者更高。数据显示，在两种学习情境中，“程序性”问题都占主导地位，但当学生准备总结性评估时更为明显。这些结果为使用LLM对学生成问题进行分类提供了基础。然而，我们识别出在使用方案进行分类能力和此类分类的价值方面存在明显的局限性：方案有限，因此难以容纳复合提示的语义丰富性，只能部分理解聊天机器人整合的更广泛风险和益处。未来，我们建议一种分析方法，能够捕捉对话的细致、多轮性质，例如通过应用来自话语心理学的对话分析方法。
英文摘要：
Providing scaffolding through educational chatbots built on Large Language Models (LLM) has potential risks and benefits that remain an open area of research. When students navigate impasses, they ask for help by formulating impasse-driven questions. Within interactions with LLM chatbots, such questions shape the user prompts and drive the pedagogical effectiveness of the chatbot's response. This paper focuses on such student questions from two datasets of distinct learning contexts: formative self-study, and summative assessed coursework. We analysed 6,113 messages from both learning contexts, using 11 different LLMs and three human raters to classify student questions using four existing schemas. On the feasibility of using LLMs as raters, results showed moderate-to-good inter-rater reliability, with higher consistency than human raters. The data showed that 'procedural' questions predominated in both learning contexts, but more so when students prepare for summative assessment. These results provide a basis on which to use LLMs for classification of student questions. However, we identify clear limitations in both the ability to classify with schemas and the value of doing so: schemas are limited and thus struggle to accommodate the semantic richness of composite prompts, offering only partial understanding the wider risks and benefits of chatbot integration. In the future, we recommend an analysis approach that captures the nuanced, multi-turn nature of conversation, for example, by applying methods from conversation analysis in discursive psychology.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
