Pre-trained language models (PLMs) are shown to be vulnerable to minor word changes, which poses a big threat to real-world systems. While previous studies directly focus on manipulating word inputs, they are limited by their means of generating adversarial samples, lacking generalization to versatile real-world attack. This paper studies the basic structure of transformer-based PLMs, the self-attention (SA) mechanism. (1) We propose a powerful perturbation technique \textit{HackAttend}, which perturbs the attention scores within the SA matrices via meticulously crafted attention masks. We show that state-of-the-art PLMs fall into heavy vulnerability that minor attention perturbations $(1\%)$ can produce a very high attack success rate $(98\%)$. Our paper expands the conventional text attack of word perturbations to more general structural perturbations. (2) We introduce \textit{S-Attend}, a novel smoothing technique that effectively makes SA robust via structural perturbations. We empirically demonstrate that this simple yet effective technique achieves robust performance on par with adversarial training when facing various text attackers. Code is publicly available at \url{github.com/liongkj/HackAttend}.
Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis
School of Computer Science & Engineering, Key Lab of New Generation Artificial Intelligence Technology & Its Interdisciplinary Applications (Ministry of Education), Southeast University, China(计算机科学与工程学院,新一代人工智能技术及跨学科应用重点实验室,东南大学,中国);Faculty of IT, Monash University, Australia(信息科技学院,墨尔本大学,澳大利亚);School of Computer Science & Engineering, Southeast University, China(计算机科学与工程学院,东南大学,中国)
From Hallucinations to Jailbreaks: Rethinking the Vulnerability of Large Foundation Models
School of Information Sciences University of Illinois at Urbana-Champaign(信息科学学院伊利诺伊大学厄巴纳-香槟分校);Computer Science and Engineering HKUST(计算机科学与工程香港科技大学);Computer Science Department University of California, Los Angeles(计算机科学系加州大学洛杉矶分校);Research Scientist, Intel Labs(英特尔实验室研究员);School of Information Sciences University of Illinois Urbana-Champaign(信息科学学院伊利诺伊大学厄巴纳-香槟分校)
Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
Chinese University of Hong Kong(香港中文大学);Microsoft Research Asia(微软亚洲研究院);Peking University(北京大学);Hong Kong University of Science and Technology(香港科技大学)
Unveiling the Pitfalls of Knowledge Editing for Large Language Models
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);ZJU-Ant Group Joint Research Center for Knowledge Graphs, Zhejiang University(浙江大学-蚂蚁集团知识图谱联合研究中心);ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University(浙江大学杭州国际科技创新中心);Tencent(腾讯)