arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.04286cs.CL
收藏

这是谁写的?零样本LLM生成文本检测的关键是GECScore

Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScore

University of Macau(澳门大学);Harbin Institute of Technology(哈尔滨工业大学)

Junchao Wu, Runzhe Zhan, Derek F. Wong, Shu Yang, Xuebo Liu, Lidia S. Chao, Min Zhang

AI总结:

本文提出一种黑盒零样本检测方法GECScore,利用人类文本比LLM生成文本包含更多语法错误的特性进行区分,在XSum和Writing Prompts数据集上平均AUROC达98.62%,优于现有SOTA方法且抗改写攻击。

摘要:

针对大语言模型(LLM)生成文本的检测器效能,在很大程度上依赖于大规模训练数据的可用性。然而,不需要此类数据的白盒零样本检测器,却受到LLM生成文本源模型可获取性的限制。在本文中,我们提出了一种简单而有效的黑盒零样本检测方法,该方法基于这样一种观察:从LLM的视角来看,人类编写的文本通常比LLM生成的文本包含更多的语法错误。该方法通过计算给定文本的语法错误纠正得分(GECScore),来区分人类编写的文本与LLM生成的文本。实验结果表明,我们的方法优于当前最先进的(SOTA)零样本和有监督方法,在XSum和Writing Prompts数据集上实现了98.62%的平均AUROC。此外,该方法在真实环境中展现出极强的可靠性,具备稳健的泛化能力以及对改写攻击的抵抗力。数据和代码可在以下地址获取:https://github.com/NLP2CT/GECScore。

英文摘要:

The efficacy of detectors for texts generated by large language models (LLMs) substantially depends on the availability of large-scale training data. However, white-box zero-shot detectors, which require no such data, are limited by the accessibility of the source model of the LLM-generated text. In this paper, we propose a simple yet effective black-box zero-shot detection approach based on the observation that, from the perspective of LLMs, human-written texts typically contain more grammatical errors than LLM-generated texts. This approach involves calculating the Grammar Error Correction Score (GECScore) for the given text to differentiate between human-written and LLM-generated text. Experimental results show that our method outperforms current state-of-the-art (SOTA) zero-shot and supervised methods, achieving an average AUROC of 98.62% across XSum and Writing Prompts dataset. Additionally, our approach demonstrates strong reliability in the wild, exhibiting robust generalization and resistance to paraphrasing attacks. Data and code are available at: https://github.com/NLP2CT/GECScore.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑