arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.23682cs.HCcs.AI
收藏

人工智能时代评估设计:一种识别人类与聊天机器人系统存在系统性反应差异的 方法

Assessment Design in the AI Era: A Method for Identifying Items Functioning Differentially for Humans and Chatbots

Weizmann Institute of Science(魏茨曼科学研究院)

Licol Zeinfeld, Alona Strugatski, Ziva Bar-Dov, Ron Blonder, Shelley Rap, Giora Alexandron

AI总结:

本文提出一种结合教育数据挖掘与心理测量理论的方法,用于识别人类与AI在评估项目上的系统性差异,以改进评估设计的公平性和有效性。

摘要:

大规模语言模型(LLMs)在教育领域的快速应用给评估设计带来了深刻挑战。为了适应LLM工具的存在,需要以可推广、有效和可靠的方式描述LLMs的优势和劣势。然而,当前LLM评估往往依赖于基准测试的描述性统计,很少有研究应用理论导向的测量方法来描述LLMs相对于人类学习者的能力,从而直接支持评估设计。本文结合教育数据挖掘和心理测量理论,提出一种统计上严谨的方法,用于识别人类和LLMs在项目上的系统性反应差异,指出评估可能最易受AI滥用的地方,以及哪些任务维度使问题对生成式AI特别容易或困难。该方法基于差异项目功能(DIF)分析——传统上用于检测不同人口群体间的偏见——结合负对照分析和项目-总分相关性辨别分析。该方法在人类学习者和六个领先聊天机器人(ChatGPT-4o及5.2,Gemini 1.5及3 Pro,Claude 3.5及4.5 Sonnet)对两套仪器(高中化学诊断测试和大学入学考试)的响应上进行了评估。领域专家随后分析了DIF标记的项目,以确定与聊天机器人超常或不足表现相关的任务维度。结果表明,DIF指导的分析提供了一个稳健的框架,用于理解LLM和人类能力的差异,并突显了其在人工智能时代改进有效、可靠和公平评估设计的价值。

英文摘要:

The rapid adoption of large language models (LLMs) in education raises profound challenges for assessment design. To adapt assessments to the presence of LLM-based tools, it is crucial to characterize the strengths and weaknesses of LLMs in a generalizable, valid and reliable manner. However, current LLM evaluations often rely on descriptive statistics derived from benchmarks, and little research applies theory-grounded measurement methods to characterize LLM capabilities relative to human learners in ways that directly support assessment design. Here, by combining educational data mining and psychometric theory, we introduce a statistically principled approach for identifying items on which humans and LLMs show systematic response differences, pinpointing where assessments may be most vulnerable to AI misuse, and which task dimensions make problems particularly easy or difficult for generative AI. The method is based on Differential Item Functioning (DIF) analysis -- traditionally used to detect bias across demographic groups -- together with negative control analysis and item-total correlation discrimination analysis. It is evaluated on responses from human learners and six leading chatbots (ChatGPT-4o \& 5.2, Gemini 1.5 \& 3 Pro, Claude 3.5 \& 4.5 Sonnet) to two instruments: a high school chemistry diagnostic test and a university entrance exam. Subject-matter experts then analyzed DIF-flagged items to characterize task dimensions associated with chatbot over- or under-performance. Results show that DIF-informed analytics provide a robust framework for understanding where LLM and human capabilities diverge, and highlight their value for improving the design of valid, reliable, and fair assessment in the AI era.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑