WETBench：用于检测维基百科上特定任务机器生成文本的基准
WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia
AI总结：
提出WETBench，一个多语言、多生成器、任务特定的基准，用于检测维基百科编辑场景下的机器生成文本，实验表明训练型检测器平均准确率78%，零样本检测器平均58%。
摘要：
鉴于维基百科作为高质量、可靠内容的可信来源，对其平台上由大型语言模型（LLM）产生的低质量机器生成文本（MGT）的扩散担忧日益增加。因此，可靠的MGT检测至关重要。然而，现有工作主要在通用生成任务上评估MGT检测器，而非维基百科编辑者更常执行的任务。这种错位可能导致在真实维基百科场景中应用时泛化能力差。我们引入了WETBench，一个多语言、多生成器、任务特定的MGT检测基准。我们定义了三个编辑任务，这些任务基于维基百科编辑者对LLM辅助编辑的感知用例进行实证：段落写作、摘要和文本风格迁移，我们使用两个新数据集在三种语言中实现这些任务。对于每个写作任务，我们评估三个提示，使用表现最佳的提示跨多个生成器生成MGT，并对多种检测器进行基准测试。我们发现，在各种设置下，基于训练的检测器平均准确率达到78%，而零样本检测器平均为58%。这些结果表明，检测器在现实生成场景中难以应对MGT，并强调了在多样化、任务特定数据上评估此类模型以评估其在编辑驱动场景中可靠性的重要性。
英文摘要：
Given Wikipedia's role as a trusted source of high-quality, reliable content, concerns are growing about the proliferation of low-quality machine-generated text (MGT) produced by large language models (LLMs) on its platform. Reliable detection of MGT is therefore essential. However, existing work primarily evaluates MGT detectors on generic generation tasks rather than on tasks more commonly performed by Wikipedia editors. This misalignment can lead to poor generalisability when applied in real-world Wikipedia contexts. We introduce WETBench, a multilingual, multi-generator, and task-specific benchmark for MGT detection. We define three editing tasks, empirically grounded in Wikipedia editors' perceived use cases for LLM-assisted editing: Paragraph Writing, Summarisation, and Text Style Transfer, which we implement using two new datasets across three languages. For each writing task, we evaluate three prompts, generate MGT across multiple generators using the best-performing prompt, and benchmark diverse detectors. We find that, across settings, training-based detectors achieve an average accuracy of 78%, while zero-shot detectors average 58%. These results show that detectors struggle with MGT in realistic generation scenarios and underscore the importance of evaluating such models on diverse, task-specific data to assess their reliability in editor-driven contexts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
