基于扩展ICAP框架的协作对话认知投入度测量：人类标注、上下文学习与反思型大语言模型智能体的比较
Measuring Cognitive Engagement in Collaborative Discourse with an Extended ICAP Framework: Comparing Human Annotation, In-Context Learning, and Reflective LLM Agents
浏览论文内容
中文总结 AI 辅助
本研究用扩展7点ICAP框架测量协作对话认知投入度，对比人类标注、ICL及反思型LLM智能体，发现人类标注信度更高，智能体方法具潜力，需强化人类标注与LLM方法的交互。
中文摘要 AI 辅助
协作可支持学习与问题解决，但其有效性取决于对话过程中的认知投入度。本研究基于交互（Interactive）、建构（Constructive）、主动（Active）、被动（Passive）模式，应用扩展的7点ICAP框架来刻画协作对话中认知投入度的变化。投入度由经训练的人类标注者编码，并与基于大语言模型（LLM）的标注方法进行比较，包括上下文学习（ICL）、零样本提示和自反思智能体。人类标注者间的信度在框架优化阶段表现稳健（kappa=0.906-0.998），高于基于ICL的标注所观察到的中等一致性（kappa=0.541-0.609）。人类优化后的框架提升了人类标注者间的一致性（Delta kappa=0.10），但仅为基于ICL的LLMs带来了适度提升（Delta kappa小于0.04）。智能体优化后的框架改善了跨模型一致性，但仍低于人类优化的框架。这些发现凸显了基于智能体方法的潜力，以及未来工作中理论引导的人类标注与基于LLM的方法持续交互的重要性。
英文摘要
Collaboration supports learning and problem-solving, but its effectiveness depends on cognitive engagement during discourse. This study applies an extended 7-point ICAP framework based on the Interactive, Constructive, Active, and Passive modes to characterize variation in cognitive engagement during collaborative dialogue. Engagement was coded by trained human annotators and compared with large language model (LLM)-based labeling approaches, including in-context learning (ICL), zero-shot prompting, and self-reflective agents. Interrater reliability among human annotators was robust across framework refinement stages (kappa = 0.906-0.998), higher than the moderate agreement observed for ICL-based annotation (kappa = 0.541-0.609). The human-refined framework improved agreement among human annotators (Delta kappa = 0.10), but produced only modest gains for ICL-based LLMs (Delta kappa less than 0.04). Agent-refined frameworks improved cross-model agreement but remained below the human-refined framework. These findings highlight the promise of agent-based approaches and the importance of continued interaction between theory-guided human annotation and LLM-based methods in future work.
发表机构
- Tufts University（塔夫茨大学）
机构由 AI 辅助整理，请以论文原文为准。
