arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2508.08096cs.CLcs.LG
收藏

LLM文本检测器在教育中的局限性

Limits of LLM Text Detectors in Education

Faculty of Technology, Bielefeld University(技术学院,比勒菲尔德大学)

Lukas Gehring, Benjamin Paaßen

AI总结:

本文提出贡献感知评估框架和GEDE基准,揭示现有LLM文本检测器在中间学生贡献等级(尤其LLM辅助修订)上分类困难,存在虚假指控风险,不适合支持教育政策执行。

摘要:

学生们越来越多地在学术写作中使用大型语言模型(LLM)的辅助。虽然轻微的辅助(如语法和风格修正,以及反馈)在大多数机构政策下是被允许的,但通常禁止将整个写作任务外包给LLM。不幸的是,当前针对LLM生成文本的检测方法主要假设人类撰写文本与LLM生成文本之间存在二元区分,忽略了现实人机协作实践的广泛性,限制了检测系统在教育评估中的有效性。在本文中,我们提出了一种面向教育的基于LLM的检测系统的贡献感知评估框架。我们引入了一个包含八个学生贡献等级的尺度,该尺度模拟了从完全人类撰写文本到LLM辅助修订,再到完全LLM生成和对抗性人性化文本的现实写作场景。机构关于LLM使用的政策随后可以转化为该尺度上可接受的LLM辅助阈值。我们进一步提出了教育中的生成性论文检测(GEDE),这是一个新颖的基准数据集,包含超过900篇人类撰写的论文和超过12,500篇生成的论文,涵盖886个任务和所有贡献等级。利用该基准,我们对四种最先进的零样本和监督检测方法进行了系统评估,跨越政策边界、贡献等级、生成模型、分布外数据和文本长度。我们表明,大多数检测器在准确分类中间学生贡献等级的文本方面存在困难,尤其是人类撰写文本的LLM辅助修订。此类错误构成了虚假指控的重大风险,表明当前的文本检测系统(仍然)不适合可靠地支持教育中关于LLM辅助的机构政策的执行。

英文摘要:

Students increasingly use the assistance of large language models (LLMs) in their academic writing. While slight assistance (e.g., grammar and style correction, as well as feedback) is permitted under most institutional policies, it is usually forbidden to offload entire writing tasks to LLMs. Unfortunately, current approaches to LLM-generated text detection predominantly assume a binary distinction between human-written and LLM-generated text, ignoring the breadth of realistic human-AI collaboration practices and limiting the validity of detection systems for educational assessment. In this paper, we propose a contribution-aware evaluation framework for LLM-based detection systems in education. We introduce a scale of eight student contribution levels that model realistic writing scenarios ranging from fully human-written texts to LLM-assisted revisions to fully LLM-generated and adversarially humanized texts. Institutional policies regarding LLM use can then be translated to thresholds of acceptable LLM assistance on this scale. We further present Generative Essay Detection in Education (GEDE), a novel benchmark dataset comprising more than 900 human-written and over 12,500 generated essays across 886 tasks and all contribution levels. Using this benchmark, we conduct a systematic evaluation of four state-of-the-art zero-shot and supervised detection methods across policy boundaries, contribution levels, generative models, out-of-distribution data, and text length. We show that most detectors struggle to accurately classify texts at intermediate student contribution levels, in particular LLM-assisted revisions of human-written texts. Such errors pose a substantial risk of false accusations, indicating that current text detection systems are (still) unsuitable to reliably support the enforcement of institutional policies regarding LLM assistance in education.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑