发表机构
Stanford University（斯坦福大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究针对基于LLM的学生话语测量的认知排斥问题，通过多民族志方法让青少年参与研究，发现学生解读与LLM测量存在不一致，凸显青少年参与认知过程的必要性。
AI 中文摘要
大语言模型（LLM）正被越来越多地用于大规模测量学生话语的各个方面，例如对话策略（talk moves）、协作、发声公平性等。通常，基于LLM的学生话语测量仅使用仅包含口头贡献的课堂对话转录文本，这会使学生语言脱离语境。验证这些测量的常见做法包括将输出与成人专家标注进行比较、使用保留的评估集以及F1分数。我们认为，这些方法不足以确保此类测量对教学和学习具有意义和公平性，尤其是对于种族和语言上处于边缘地位的青少年而言。为了以所分析话语的青少年为中心，重新构建这些课堂对话的语境并让青少年参与研究过程是必要的。与青少年共享认知权威，最终会以他们的观点为中心，并为分析他们的话语增添成人专家、研究人员和LLMs无法提供的关键细微差别。在对一所八年级数学课堂中的多语言青少年进行的案例研究中，我们采用多种民族志导向的方法来重新构建学生对话的语境，并让青少年作为认知权威与研究人员和LLMs互动，以此解决青少年的认知排斥问题。我们对四名核心学生进行了参与式观察、访谈、焦点小组讨论以及成员检验。研究结果显示，学生对自身数学话语经历的解读与基于LLM的话语测量之间存在不一致。学生对LLM分类和用于测量其话语的编码方案均提出质疑，凸显了让青少年参与生成关于自身经历知识的认知过程的必要性。
英文摘要
LLMs are being used increasingly to measure aspects of student discourse (e.g. talk moves, collaboration, equity of voice) at scale. Typically, LLM-based measures of student talk use transcriptions of classroom conversations that only include verbal contributions, which de-contextualize student language. Common practices for validating these measures include comparing outputs against expert annotations by adults, using held out evaluation sets and F1 scores. We argue that these approaches are insufficient to ensure that such measures are meaningful and equitable for teaching and learning, particularly for racially and linguistically marginalized youth. In order to center the youth whose talk is being analyzed, re-contextualizing these classroom conversations and engaging youth in the research process is necessary. Sharing epistemic authority with youth, ultimately, centers their point of view and adds crucial nuance to the analysis of their talk that adult experts, researchers, and LLMs cannot provide. In a case study of multilingual youth in one 8th-grade math classroom, we address the epistemic exclusion of youth by employing multiple ethnographically-oriented methods to re-contextualize student conversations and center youth as epistemic authorities in conversation with researchers and LLMs. We conducted participant observations, interviews, focus groups, and member checks with four focal students. Findings reveal that there were misalignments between students' interpretations of their own math talk experiences and the LLM-based measures of their talk. Students contested both the LLM classifications and the coding scheme used to measure their talk, highlighting the need for youth to be involved in the epistemic process of producing knowledge about their experiences.
Commentsunder review
