arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.13353cs.AI
收藏

通过多智能体协调优化LLM对课堂话语的标注

Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration

Cornell University(康奈尔大学)

Bakhtawar Ahtisham, Kirk Vanacore, Rene F. Kizilcec

AI总结:

本文提出一种分层、成本敏感的多阶段标注框架,通过自检和争议解决阶段提升LLM标注的可靠性,以应对教育数据科学中规模与效度之间的矛盾。

摘要:

大型语言模型(LLM)正被越来越多地视为标注教育数据(包括课堂话语、互动日志和定性学习成果)的可扩展工具。其能够快速总结教学互动并分配符合评分标准的标签,已推动了减少专家人工标注成本和时间的乐观预期。然而,越来越多的证据表明,单次LLM输出仍无法可靠地用于需要上下文、教学或规范判断的高风险教育构念,如教学意图或话语步骤。这种规模与效度之间的张力是当代教育数据科学的核心问题。在本文中,我们提出并经验性评估了一种分层、成本敏感的协调框架,用于LLM标注,以在提高可靠性的同时显式建模计算权衡。与其将标注视为一次预测问题,我们将其概念化为一个多阶段知识过程,包括(1)未经验证的单次标注阶段,其中模型根据评分标准独立分配标签;(2)自我验证阶段,其中每个模型检查其输出是否符合评分标准定义,并在发现不一致时修订其标签;(3)以争议为中心的裁决阶段,其中独立裁决模型检查已验证的标签和论证,并根据评分标准确定最终标签。这种结构与教育研究中已建立的人类标注工作流程相呼应,其中初始编码后进行自我检查,并由专家解决争议。

英文摘要:

Large language models (LLMs) are increasingly positioned as scalable tools for annotating educational data, including classroom discourse, interaction logs, and qualitative learning artifacts. Their ability to rapidly summarize instructional interactions and assign rubric-aligned labels has fueled optimism about reducing the cost and time associated with expert human annotation. However, growing evidence suggests that single-pass LLM outputs remain unreliable for high-stakes educational constructs that require contextual, pedagogical, or normative judgment, such as instructional intent or discourse moves. This tension between scale and validity sits at the core of contemporary education data science. In this work, we present and empirically evaluate a hierarchical, cost-aware orchestration framework for LLM-based annotation that improves reliability while explicitly modeling computational tradeoffs. Rather than treating annotation as a one-shot prediction problem, we conceptualize it as a multi-stage epistemic process comprising (1) an unverified single-pass annotation stage, in which models independently assign labels based on the rubric; (2) a self-verification stage, in which each model audits its own output against rubric definitions and revises its label if inconsistencies are detected; and (3) a disagreement-centric adjudication stage, in which an independent adjudicator model examines the verified labels and justifications and determines a final label in accordance with the rubric. This structure mirrors established human annotation workflows in educational research, where initial coding is followed by self-checking and expert resolution of disagreements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑