Models trained via Contrastive Language-Image Pretraining (CLIP) serve as the foundational vision encoders for most modern Large Vision Language Models (LVLMs). Despite their widespread adoption, CLIP models exhibit a critical yet underexplored failure mode: irrelevant text appearing within images confounds visual representations, biasing them toward lexical meaning rather than true visual semantics. This robustness issue, commonly described as a Typographic Attack (TA), exposes a vulnerability that poses a significant risk to safety-critical applications such as autonomous driving. To achieve interpretable and effective robustness against TA, we propose a novel, training-free mechanistic interpretability method. Our method provides sampling-based interpretations of hidden state representations and quantitatively attributes semantic versus lexical focus to individual attention heads. Through probabilistic analysis and circuit mining, we isolate specific Vision Transformer (ViT) components that disproportionately encode lexical information, thereby identifying the mechanistic source of TA. We further show that simple interventions applied directly to the identified circuits, without any additional training, can substantially improve robustness against Typographic Attacks in object classification. These interventions, such as selective adjustment of attention weights, also outperform both supervised and training-free defense methods. Our experiments demonstrate that applying the proposed intervention to the vision encoders of several state-of-the-art LVLMs yields substantial gains in Visual Question Answering accuracy under Typographic Attack interference on RIO-Bench. These results confirm both the efficacy and the generalizability of our mechanistic approach. Code is released at https://github.com/Liu-524/SamplingTAR.
Unveiling Typographic Deceptions: Insights of the Typographic Vulnerability in Large Vision-Language Model
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));University of Oxford(牛津大学);Xi’an Jiaotong University(西安交通大学);Drexel University(德雷塞尔大学);The Hong Kong University of Science and Technology(香港科技大学)
Dyslexify: A Mechanistic Defense Against Typographic Attacks in CLIP
Fraunhofer Heinrich Hertz Institute(弗劳恩霍夫海因里希·赫兹研究所);University of Oxford(牛津大学);Technological University Dublin(都柏林技术大学);Technische Universität Berlin(柏林技术大学)
SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Models
BLISS e.V.(BLISS协会);Berliner Hochschule für Technik (BHT)(柏林技术大学);Technische Universität Berlin(柏林技术大学);KI Werkstatt, Hochschule für Technik und Wirtschaft Berlin (HTW)(柏林技术经济学院);Merantix Momentum(Merantix Momentum公司);Fraunhofer Heinrich-Hertz-Institut, Berlin, Germany(柏林弗劳恩霍夫 Heinrich-Hertz 研究所)
SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environments
Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);University of Alberta(阿尔伯塔大学);Tianjin University(天津大学)
FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts
Tsinghua University(清华大学);Carnegie Mellon University(卡内基梅隆大学);Zhongguancun Laboratory(中关村实验室);National Financial Cryptography Research Center(国家金融密码学研究中心);Shandong Institute of Blockchain(山东区块链研究院);Shandong University(山东大学)
Beyond Pixels: Semantic-aware Typographic Attack for Geo-Privacy Protection
Hangzhou Institute of Technology, Xidian University, China(杭州科技学院,西安电子科技大学,中国);National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学);Nankai University(南开大学);New York University(纽约大学);East China Normal University(华东师范大学);Hikvision Digital Technology Co., Ltd(海康威视数字技术有限公司);Shanghai Industrial Control Safety Innovation Tech. Co., Ltd(上海工业控制安全创新技术有限公司)
Physical Prompt Injection Attacks on Large Vision-Language Models
School of Cyber Science and Engineering, Wuhan University(武汉大学计算机科学与工程学院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics(南京航空航天大学计算机科学与技术学院)
Transferable Adversarial Attack on Image Tampering Localization
Communication University of China(中国传媒大学);State Key Laboratory of Media Convergence and Communication, Communication University of China(中国传媒大学媒体融合与传播国家重点实验室);School of Computer and Cyber Sciences, Communication University of China(中国传媒大学计算机与网络空间安全学院)
Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models
Mohamed Bin Zayed University of AI, UAE(穆罕默德·本·扎耶德人工智能大学,阿联酋);Khalifa University, UAE(卡比拉大学,阿联酋);Australian National University, Australia(澳大利亚国立大学,澳大利亚)
Privacy-Preserving Representations are not Enough -- Recovering Scene Content from Camera Poses
Chalmers University of Technology(查尔姆斯理工大学);Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague(捷克布拉格技术大学捷克信息学、机器人学与控制论研究所);Faculty of Electrical Engineering, Czech Technical University in Prague(捷克布拉格技术大学电气工程学院)
ForensicsSAM: Toward Robust and Unified Image Forgery Detection and Localization Resisting to Adversarial Attack
Shenzhen Key Laboratory of Media Security, Faculty of Electronic and Information Engineering, Shenzhen University, China(深圳媒体安全重点实验室,电子与信息工程学院,深圳大学,中国);Rapid-Rich Object Search (ROSE) Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore(快速富对象搜索(ROSE)实验室,电气电子工程学院,南洋理工大学,新加坡);Guangdong Laboratory of Machine Perception and Intelligent Computing, Faculty of Engineering, Shenzhen MSU-BIT University, China(广东机器感知与智能计算实验室,工程学院,深圳MSU-BIT大学,中国)
Active Adversarial Noise Suppression for Image Forgery Localization
College of Electronic and Information Engineering, Shenzhen University(深圳大学电子与信息工程学院);Shenzhen Key Laboratory of Media Security(深圳媒体安全重点实验室);Guangdong Laboratory of Machine Perception and Intelligent Computing, Faculty of Engineering, Shenzhen MSU-BIT University(广东机器感知与智能计算实验室,深圳MSU-BIT大学工程学院)
Adversarial Attacks and Detection in Visual Place Recognition for Safer Robot Navigation
QUT Centre for Robotics(昆士兰科技大学机器人中心);School of Electrical Engineering and Robotics at the Queensland University of Technology(昆士兰科技大学电气工程与机器人学院);CIICADA Lab(CIICADA实验室)
Jedi: Entropy-based Localization and Removal of Adversarial Patches
Université de Sousse(苏塞大学);Ecole Nationale d’Ingénieurs de Sousse(苏塞国家工程师学院);University of California Riverside(加州大学河滨分校);Queen’s University Belfast(贝尔法斯特女王大学);CNRS(法国国家科学研究中心);Université Polytechnique Hauts-de-France(上法兰西综合理工大学);Université de Jendouba(坚杜巴大学);Institut National des Technologies et des Sciences du Kef(卡夫国家技术与科学学院)