CopyBench：衡量语言模型生成中受版权保护文本的字面与非字面复制
CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation
AI总结：
本文提出 CopyBench 基准，首次同时衡量语言模型生成中的字面与非字面版权复制，发现大模型复制更严重，且现有缓解策略对非字面复制效果有限。
摘要：
评估语言模型（LMs）对受版权保护内容的复制程度，对人工智能和法律界都具有重要意义。尽管法院在评估复制程度时会同时考虑字面和非字面相似性，但先前的研究仅关注字面相似性。为弥补这一空白，我们引入了 CopyBench，这是一个旨在衡量语言模型生成中字面和非字面复制行为的基准。我们以受版权保护的小说书籍作为文本来源，提供了自动评估协议，用于评估字面和非字面复制，同时兼顾模型在回忆受版权作品中的事实和生成流畅续写方面的实用性。我们发现，尽管字面复制相对罕见，但两种类型的非字面复制——事件复制和角色复制——即使在参数规模小至 7B 的模型中也会出现。更大的模型表现出显著更多的复制行为，在比较 Llama3-8B 和 70B 模型时，字面复制率从 0.2% 增加到 10.5%，非字面复制率从 2.3% 增加到 5.9%。我们进一步评估了当前缓解复制策略的有效性，并表明：（1）训练时对齐可以减少字面复制，但可能增加非字面复制；（2）当前的推理时缓解方法主要减少字面复制，而无法减少非字面复制。
英文摘要：
Evaluating the degree of reproduction of copyright-protected content by language models (LMs) is of significant interest to the AI and legal communities. Although both literal and non-literal similarities are considered by courts when assessing the degree of reproduction, prior research has focused only on literal similarities. To bridge this gap, we introduce CopyBench, a benchmark designed to measure both literal and non-literal copying in LM generations. Using copyrighted fiction books as text sources, we provide automatic evaluation protocols to assess literal and non-literal copying, balanced against the model utility in terms of the ability to recall facts from the copyrighted works and generate fluent completions. We find that, although literal copying is relatively rare, two types of non-literal copying -- event copying and character copying -- occur even in models as small as 7B parameters. Larger models demonstrate significantly more copying, with literal copying rates increasing from 0.2\% to 10.5\% and non-literal copying from 2.3\% to 5.9\% when comparing Llama3-8B and 70B models, respectively. We further evaluate the effectiveness of current strategies for mitigating copying and show that (1) training-time alignment can reduce literal copying but may increase non-literal copying, and (2) current inference-time mitigation methods primarily reduce literal but not non-literal copying.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
