带后门的预训练语言模型水印算法
Watermarking Pre-trained Language Models with Backdooring
摘要：
大型预训练语言模型（PLM）已被证明是现代自然语言处理系统的重要组成部分。PLM通常需要在特定于任务的下游数据集上进行微调，由于灾难性遗忘现象，这使得很难主张PLM的所有权并保护开发人员的知识产权。我们表明，PLM可以通过嵌入由所有者定义的特定输入触发的后门来使用多任务学习框架进行水印，即使水印PLM在多个下游任务上进行了微调，这些水印也很难去除。除了使用一些罕见的单词作为触发器，我们还表明，常见的单词的组合可以作为后门触发器，以避免他们被轻易地检测到。在多个数据集上的实验表明，该算法能够鲁棒地提取出嵌入的水印，成功率高，受后续微调的影响较小。
英文摘要：
Large pre-trained language models (PLMs) have proven to be a crucial component of modern natural language processing systems. PLMs typically need to be fine-tuned on task-specific downstream datasets, which makes it hard to claim the ownership of PLMs and protect the developer's intellectual property due to the catastrophic forgetting phenomenon. We show that PLMs can be watermarked with a multi-task learning framework by embedding backdoors triggered by specific inputs defined by the owners, and those watermarks are hard to remove even though the watermarked PLMs are fine-tuned on multiple downstream tasks. In addition to using some rare words as triggers, we also show that the combination of common words can be used as backdoor triggers to avoid them being easily detected. Extensive experiments on multiple datasets demonstrate that the embedded watermarks can be robustly extracted with a high success rate and less influenced by the follow-up fine-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
