文本水印家族的下游权衡
Downstream Trade-offs of a Family of Text Watermarks
AI总结：
本文评估了三种LLM文本水印策略在分类、问答、短文本和长文本生成等任务上的下游影响，发现水印会导致模型效用显著下降，揭示了使用水印模型时的性能权衡。
摘要：
水印涉及在生成的文本中植入一种不易察觉的信号，该信号之后可通过统计测试被检测出来。一类面向大语言模型（LLMs）的突出水印策略通过在每一步生成时对（伪随机选择的）令牌子集进行上采样来嵌入该信号。然而，这类信号会改变模型的输出分布，并可能对其下游性能产生非预期的影响。在本工作中，我们评估了使用三种不同策略加水印的LLMs在一系列多样化任务上的表现，这些任务包括k类分类（CLS）、多项选择问答（MCQ）、短文本生成（例如开放式问答）和长文本生成（例如翻译）任务。我们发现，在现实的超参数设置下，水印会导致LLMs在所有任务上的有效效用显著下降。我们观察到，在CLS任务中平均情况下性能下降10%至20%，最坏情况下下降幅度可高达100%。我们注意到，MCQ任务的性能下降约7%，短文本生成任务下降10%至15%，长文本生成任务下降5%至15%。我们的发现强调了用户在使用带水印模型时应当意识到的权衡。
英文摘要：
Watermarking involves implanting an imperceptible signal into generated text that can later be detected via statistical tests. A prominent family of watermarking strategies for LLMs embeds this signal by upsampling a (pseudorandomly-chosen) subset of tokens at every generation step. However, such signals alter the model's output distribution and can have unintended effects on its downstream performance. In this work, we evaluate the performance of LLMs watermarked using three different strategies over a diverse suite of tasks including those cast as k-class classification (CLS), multiple choice question answering (MCQ), short-form generation (e.g., open-ended question answering) and long-form generation (e.g., translation) tasks. We find that watermarks (under realistic hyperparameters) can cause significant drops in LLMs' effective utility across all tasks. We observe drops of 10 to 20% in CLS tasks in the average case, which shoot up to 100% in the worst case. We notice degradations of about 7% in MCQ tasks, 10-15% in short-form generation, and 5-15% in long-form generation tasks. Our findings highlight the trade-offs that users should be cognizant of when using watermarked models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
