Representation Engineering (RepE) has emerged as a powerful paradigm for enhancing AI transparency by focusing on high-level representations rather than individual neurons or circuits. It has proven effective in improving interpretability and control, showing that representations can emerge, propagate, and shape final model outputs in large language models (LLMs). However, in Vision-Language Models (VLMs), visual input can override factual linguistic knowledge, leading to hallucinated responses that contradict reality. To address this challenge, we make the first attempt to extend RepE to VLMs, analyzing how multimodal representations are preserved and transformed. Building on our findings and drawing inspiration from successful RepE applications, we develop a theoretical framework that explains the stability of neural activity across layers using the principal eigenvector, uncovering the underlying mechanism of RepE. We empirically validate these instrinsic properties, demonstrating their broad applicability and significance. By bridging theoretical insights with empirical validation, this work transforms RepE from a descriptive tool into a structured theoretical framework, opening new directions for improving AI robustness, fairness, and transparency.
Why Text Prevails: Vision May Undermine Multimodal Medical Decision Making
University of Texas Rio Grande Valley(德克萨斯大学里奥格兰德谷大学);University of Pittsburgh(匹兹堡大学);NC State University(北卡罗来纳州立大学);University of Washington(华盛顿大学);University of Maryland(马里兰大学)
Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine
Department of Computer Science, City University of Hong Kong (Dongguan)(香港城市大学(东莞)计算机科学系);College of Computing and Data Science (CCDS), Nanyang Technological University(南洋理工大学计算与数据科学学院);School of Computer Science, Northwestern Polytechnical University(西北工业大学计算机科学学院)
SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(北京大学计算机科学学院多媒体信息处理国家重点实验室);University of Pennsylvania(宾夕法尼亚大学);University of Electronic Science and Technology of China(电子科技大学);Nanyang Technological University(南洋理工大学);Tsinghua University(清华大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));De Artificial Intelligence Lab(德人工智能实验室)
Agents in Software Engineering: Survey, Landscape, and Vision
Sun Yat-sen University(中山大学);Xi’an Jiaotong University(西安交通大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Xiamen University(厦门大学);Huawei Cloud Computing Technologies Co., Ltd.(华为云计算技术有限公司)
From Drawings to Decisions: A Hybrid Vision-Language Framework for Parsing 2D Engineering Drawings into Structured Manufacturing Knowledge
Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR)(新加坡制造技术研究院,新加坡科技研究局);Advanced Remanufacturing and Technology Centre (ARTC), Agency for Science, Technology and Research (A*STAR)(先进再制造与技术中心,新加坡科技研究局);Nanyang Technological University(南洋理工大学)
Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models
New York University, New York, USA(纽约大学);Tsinghua University, Beijing, China(清华大学);Columbia University, New York, USA(哥伦比亚大学);University of Michigan, Ann Arbor, USA(密歇根大学)
Fine-Tuning Vision-Language Model for Automated Engineering Drawing Information Extraction
Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局(A*STAR)新加坡制造技术研究院(SIMTech));Advanced Remanufacturing and Technology Centre (ARTC), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局(A*STAR)先进再制造与技术中心(ARTC));Nanyang Technological University(南洋理工大学)