Self-supervised learning in computer vision aims to pre-train an image encoder using a large amount of unlabeled images or (image, text) pairs. The pre-trained image encoder can then be used as a feature extractor to build downstream classifiers for many downstream tasks with a small amount of or no labeled training data. In this work, we propose BadEncoder, the first backdoor attack to self-supervised learning. In particular, our BadEncoder injects backdoors into a pre-trained image encoder such that the downstream classifiers built based on the backdoored image encoder for different downstream tasks simultaneously inherit the backdoor behavior. We formulate our BadEncoder as an optimization problem and we propose a gradient descent based method to solve it, which produces a backdoored image encoder from a clean one. Our extensive empirical evaluation results on multiple datasets show that our BadEncoder achieves high attack success rates while preserving the accuracy of the downstream classifiers. We also show the effectiveness of BadEncoder using two publicly available, real-world image encoders, i.e., Google's image encoder pre-trained on ImageNet and OpenAI's Contrastive Language-Image Pre-training (CLIP) image encoder pre-trained on 400 million (image, text) pairs collected from the Internet. Moreover, we consider defenses including Neural Cleanse and MNTD (empirical defenses) as well as PatchGuard (a provable defense). Our results show that these defenses are insufficient to defend against BadEncoder, highlighting the needs for new defenses against our BadEncoder. Our code is publicly available at: https://github.com/jjy1994/BadEncoder.
Mutual Information Guided Backdoor Mitigation for Pre-trained Encoders
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）
Can Distillation Mitigate Backdoor Attacks in Pre-trained Encoders?
State Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；Shenzhen Research Institute of Nanjing University（南京大学深圳研究院）；School of Computer Science and Engineering, University of New South Wales（新南威尔士大学计算机科学与工程学院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）
GhostEncoder: Stealthy Backdoor Attacks with Dynamic Triggers to Pre-trained Encoders in Self-supervised Learning
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Nanjing University of Aeronautics and Astronautics Shenzhen Research Institute（南京航空航天大学深圳研究院）；Zhejiang Lab（之江实验室）；Wuhan University（武汉大学）；Xi’an Jiaotong University（西安交通大学）
Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesis
Technical University of Darmstadt（达姆施塔特工业大学）；Centre for Cognitive Science（认知科学中心）；Hessian Center for AI (hessian.AI)（黑森人工智能中心（hessian.AI））；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））
Backdoor Attacks on Prompt-Driven Video Segmentation Foundation Models
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Fujian Normal University（福建师范大学）；Jinan University（暨南大学）
German Research Center for Artificial Intelligence（德国人工智能研究中心）；Technical University of Darmstadt（达姆施塔特工业大学）；TU Dortmund University（多特蒙德工业大学）；Center for Trustworthy Data Science and Security, University Alliance Ruhr（鲁尔大学联盟可信数据科学与安全中心）；Hessian Center for AI (hessian.AI)（黑森州人工智能中心）；Centre for Cognitive Science TU Darmstadt（达姆施塔特工业大学认知科学中心）
Tsinghua University（清华大学）；Shandong University（山东大学）；Wuhan University（武汉大学）；Zhongguancun Laboratory（中关村实验室）；Shandong Institute of Blockchain（山东省区块链研究院）；National Financial Cryptography Research Center（国家金融密码研究中心）
HPE: Hallucinated Positive Entanglement for Backdoor Attacks in Federated Self-Supervised Learning
School of Information Engineering, Yangzhou University（扬州大学信息工程学院）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Computing and Artificial Intelligence, Chaohu University（巢湖学院计算与人工智能学院）；College of Data Science, Jiaxing University（嘉兴大学数据科学学院）；Tacoma School of Engineering and Technology, University of Washington（华盛顿大学塔科马工程技术学院）
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Nanyang Technological University（南洋理工大学）；Sun Yat-sen University-Shenzhen（中山大学深圳校区）
Robust and Transferable Backdoor Attacks Against Deep Image Compression With Selective Frequency Prior
Nanyang Technological University（南洋理工大学）；Pengcheng Laboratory（鹏城实验室）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；School of Computer Science, Peking University（北京大学计算机学院）
SSL-WM: A Black-Box Watermarking Approach for Encoders Pre-trained by Self-supervised Learning
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Department of Computer Science, Metropolitan College, Boston University（波士顿大学大都会学院计算机科学系）
