TURINGBENCH：神经文本生成时代的图灵测试基准环境
TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation
摘要：
生成语言模型的最新进展使机器能够生成极其逼真的文本。虽然这种模型有许多合法的应用，但也越来越需要区分机器生成的文本和人类书写的文本（例如，假新闻检测）。然而，据我们所知，目前还没有具有数据集和任务的基准环境来系统地研究神经文本生成方法的所谓“图灵测试”问题。在这项工作中，我们提出了TuringBench基准测试环境，它由（1）一个数据集，其中包含20万个人类或机器生成的样本，跨越20个标签{人类，GPT-1，GPT-2_small，GPT-2_medium，GPT-2_large，GPT-2_xl，GPT-2_PyTorch，GPT-3，GROVER_base，GROVER_large，GROVER_mega，CTRL，XLM，XLNET_base，XLNET_large，FAIR_wmt19，FAIR_wmt20，（2）两个基准任务--即，图灵测试（TT）和作者归属（AA），以及（3）一个网站与排行榜。我们使用TuringBench的初步实验结果表明，FAIR_wmt20和GPT-3是目前的赢家，在所有测试的语言模型，在生成最像人类的不可区分的文本与最低的F1分数由五个国家的最先进的TT检测模型。TuringBench可从以下网址获得：https://turingbench.ist.psu.edu/
英文摘要：
Recent progress in generative language models has enabled machines to generate astonishingly realistic texts. While there are many legitimate applications of such models, there is also a rising need to distinguish machine-generated texts from human-written ones (e.g., fake news detection). However, to our best knowledge, there is currently no benchmark environment with datasets and tasks to systematically study the so-called "Turing Test" problem for neural text generation methods. In this work, we present the TuringBench benchmark environment, which is comprised of (1) a dataset with 200K human- or machine-generated samples across 20 labels {Human, GPT-1, GPT-2_small, GPT-2_medium, GPT-2_large, GPT-2_xl, GPT-2_PyTorch, GPT-3, GROVER_base, GROVER_large, GROVER_mega, CTRL, XLM, XLNET_base, XLNET_large, FAIR_wmt19, FAIR_wmt20, TRANSFORMER_XL, PPLM_distil, PPLM_gpt2}, (2) two benchmark tasks -- i.e., Turing Test (TT) and Authorship Attribution (AA), and (3) a website with leaderboards. Our preliminary experimental results using TuringBench show that FAIR_wmt20 and GPT-3 are the current winners, among all language models tested, in generating the most human-like indistinguishable texts with the lowest F1 score by five state-of-the-art TT detection models. The TuringBench is available at: https://turingbench.ist.psu.edu/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
