arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.16288cs.CL
收藏

PlagBench:探索大型语言模型在剽窃生成与检测中的双重性

PlagBench: Exploring the Duality of Large Language Models in Plagiarism Generation and Detection

The Pennsylvania State University(宾夕法尼亚州立大学);Vellore Institute of Technology(韦洛尔理工大学);MIT Lincoln Laboratory(麻省理工学院林肯实验室);Indiana University(印第安纳大学)

Jooyoung Lee, Toshini Agrawal, Adaku Uchendu, Thai Le, Jinghui Chen, Dongwon Lee

AI总结:

本文构建含46.5K文本对的PlagBench数据集,覆盖逐字复制、释义和摘要三类剽窃,系统评估LLM在剽窃生成与检测上的双重能力。

摘要:

近期研究引发了对大型语言模型(LLM)可能给学术诚信和版权保护带来威胁的担忧。然而,这些研究主要集中于对原文的逐字复制。此外,LLM如何促进检测LLM生成的剽窃内容,在很大程度上仍未得到探索。为弥补这些空白,我们推出了PlagBench,这是一个包含46.5K个合成文本对的数据集,代表三种主要剽窃类型:逐字复制、释义和摘要。这些样本由三个先进的LLM生成。我们通过结合细粒度自动评估和人工标注,严格验证了PlagBench的质量。随后,我们将该数据集用于两个目的:(1)考察LLM将原始内容转化为准确释义和摘要的能力;(2)评估五个现代LLM以及三个专用剽窃检查工具的剽窃检测性能。结果表明,与GPT-4 Turbo相比,GPT-3.5 Turbo能够生成高质量释义和摘要,且不会显著增加文本复杂度。然而,在检测方面,GPT-4比其他LLM和商业检测工具高出20%,这凸显了LLM不仅在内容生成方面,也在剽窃检测方面不断发展的能力。数据和源代码可在https://github.com/Brit7777/plagbench获取。

英文摘要:

Recent studies have raised concerns about the potential threats large language models (LLMs) pose to academic integrity and copyright protection. Yet, their investigation is predominantly focused on literal copies of original texts. Also, how LLMs can facilitate the detection of LLM-generated plagiarism remains largely unexplored. To address these gaps, we introduce \textbf{\sf PlagBench}, a dataset of 46.5K synthetic text pairs that represent three major types of plagiarism: verbatim copying, paraphrasing, and summarization. These samples are generated by three advanced LLMs. We rigorously validate the quality of PlagBench through a combination of fine-grained automatic evaluation and human annotation. We then utilize this dataset for two purposes: (1) to examine LLMs' ability to transform original content into accurate paraphrases and summaries, and (2) to evaluate the plagiarism detection performance of five modern LLMs alongside three specialized plagiarism checkers. Our results show that GPT-3.5 Turbo can produce high-quality paraphrases and summaries without significantly increasing text complexity compared to GPT-4 Turbo. However, in terms of detection, GPT-4 outperforms other LLMs and commercial detection tools by 20%, highlights the evolving capabilities of LLMs not only in content generation but also in plagiarism detection. Data and source code are available at https://github.com/Brit7777/plagbench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑