反向图灵测试 CT^2:AI生成文本检测并非如你所想的那般简单——引入AI可检测性指数
Counter Turing Test CT^2: AI-Generated Text Detection is Not as Easy as You May Think -- Introducing AI Detectability Index
AI总结:
本文提出Counter Turing Test (CT^2)基准评估AI生成文本检测(AGTD)技术的脆弱性,并引入AI Detectability Index (ADI)对15个LLM的可检测性进行量化排名,发现较大LLM更难被检测。
摘要:
随着ChatGPT的广泛普及,AI生成文本的风险与后果已令人担忧地增加。为解决AI生成制品的归属这一不可避免的问题,美国版权局发布声明称:“如果作品的传统作者要素由机器产生,则该作品缺乏人类作者身份,版权局将不予登记。”此外,美国和欧盟政府近期已起草了关于AI监管框架的初步提案。鉴于生成式AI备受瞩目,AI生成文本检测(AGTD)已成为一个已在研究中获得即时关注的课题,一些初始方法已被提出,紧接着便出现了规避检测的技术。本文引入了Counter Turing Test (CT^2),这是一个旨在全面评估现有AGTD技术鲁棒性的基准。我们的实证发现明确凸显了受审查的AGTD方法的脆弱性。在围绕监管AI发展的政策制定进行广泛讨论之际,评估LLM生成内容的可检测性至关重要。因此,为建立一个可量化的谱系以根据可检测性水平对LLM进行评估和排名,我们提出了AI Detectability Index (ADI)。我们对15个当代LLM进行了彻底检查,实证表明较大的LLM往往具有更高的ADI,这意味着与较小的LLM相比,它们更难被检测。我们坚信ADI作为工具对更广泛的NLP社区具有重要价值,有望成为AI相关政策制定的准则。
英文摘要:
With the rise of prolific ChatGPT, the risk and consequences of AI-generated text has increased alarmingly. To address the inevitable question of ownership attribution for AI-generated artifacts, the US Copyright Office released a statement stating that 'If a work's traditional elements of authorship were produced by a machine, the work lacks human authorship and the Office will not register it'. Furthermore, both the US and the EU governments have recently drafted their initial proposals regarding the regulatory framework for AI. Given this cynosural spotlight on generative AI, AI-generated text detection (AGTD) has emerged as a topic that has already received immediate attention in research, with some initial methods having been proposed, soon followed by emergence of techniques to bypass detection. This paper introduces the Counter Turing Test (CT^2), a benchmark consisting of techniques aiming to offer a comprehensive evaluation of the robustness of existing AGTD techniques. Our empirical findings unequivocally highlight the fragility of the proposed AGTD methods under scrutiny. Amidst the extensive deliberations on policy-making for regulating AI development, it is of utmost importance to assess the detectability of content generated by LLMs. Thus, to establish a quantifiable spectrum facilitating the evaluation and ranking of LLMs according to their detectability levels, we propose the AI Detectability Index (ADI). We conduct a thorough examination of 15 contemporary LLMs, empirically demonstrating that larger LLMs tend to have a higher ADI, indicating they are less detectable compared to smaller LLMs. We firmly believe that ADI holds significant value as a tool for the wider NLP community, with the potential to serve as a rubric in AI-related policy-making.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。