arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.03508cs.LGcs.AIcs.CR
收藏

互信息引导的预训练编码器后门缓解方法

Mutual Information Guided Backdoor Mitigation for Pre-trained Encoders

State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算机与数据科学学院)

Tingxu Han, Weisong Sun, Ziqi Ding, Chunrong Fang, Hanwei Qian, Jiaxun Li, Zhenyu Chen, Xiangyu Zhang

AI总结:

针对自监督学习预训练编码器易受后门攻击、现有缓解方法因缺标签效果受限的问题,提出互信息引导的MIMIC方法,通过带互信息定位的知识蒸馏用少量干净数据高效缓解后门,性能优于七种现有最优方法。

摘要:

自监督学习(SSL)在无需标注数据的预训练编码器领域日益受到关注,基于这类预训练编码器构建的下游任务可达到接近当前最优的性能。然而现有研究表明,SSL预训练的编码器易受后门攻击。目前众多后门缓解技术都是针对下游任务模型设计的,由于预训练阶段缺乏标签信息,这些方法适配到预训练编码器时效果会受损且存在局限。为解决针对预训练编码器的后门攻击问题,本文创新性地提出一种互信息引导的后门缓解技术,名为MIMIC。MIMIC将可能存在后门的编码器作为教师网络,采用知识蒸馏从教师网络中蒸馏出干净的学生编码器。与现有知识蒸馏方法不同,MIMIC使用随机权重初始化学生网络,不会从教师网络继承任何后门。随后MIMIC利用各层与提取特征之间的互信息定位教师网络中良性知识的位置,以此开展蒸馏,将干净特征从教师网络克隆到学生网络。我们从两个方面设计蒸馏损失,包括克隆损失和注意力损失,旨在同时缓解后门并保持编码器性能。我们针对SSL中的两种后门攻击开展评估,结果表明MIMIC仅使用不到5%的干净数据就能显著降低攻击成功率,性能优于七种当前最优的后门缓解技术。

英文摘要:

Self-supervised learning (SSL) is increasingly attractive for pre-training encoders without requiring labeled data. Downstream tasks built on top of those pre-trained encoders can achieve nearly state-of-the-art performance. The pre-trained encoders by SSL, however, are vulnerable to backdoor attacks as demonstrated by existing studies. Numerous backdoor mitigation techniques are designed for downstream task models. However, their effectiveness is impaired and limited when adapted to pre-trained encoders, due to the lack of label information when pre-training. To address backdoor attacks against pre-trained encoders, in this paper, we innovatively propose a mutual information guided backdoor mitigation technique, named MIMIC. MIMIC treats the potentially backdoored encoder as the teacher net and employs knowledge distillation to distill a clean student encoder from the teacher net. Different from existing knowledge distillation approaches, MIMIC initializes the student with random weights, inheriting no backdoors from teacher nets. Then MIMIC leverages mutual information between each layer and extracted features to locate where benign knowledge lies in the teacher net, with which distillation is deployed to clone clean features from teacher to student. We craft the distillation loss with two aspects, including clone loss and attention loss, aiming to mitigate backdoors and maintain encoder performance at the same time. Our evaluation conducted on two backdoor attacks in SSL demonstrates that MIMIC can significantly reduce the attack success rate by only utilizing <5% of clean data, surpassing seven state-of-the-art backdoor mitigation techniques.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑