Identifying feature representations in language models is a central task in mechanistic interpretability. Several recent studies have made the observation that feature representations can be inferred in some cases from singular vectors of attention matrices. However, sound justification for this phenomenon is lacking. In this paper we address that question, asking: why and when do singular vectors align with features? First, we demonstrate that singular vectors robustly align with features in a model where features can be directly observed. We then show theoretically that such alignment is expected under a range of conditions. We close by asking how, operationally, alignment may be recognized in real models where feature representations are not directly observable. We identify sparse attention decomposition as a testable prediction of alignment, and show evidence that it emerges in real models in a manner consistent with predictions. Together these results suggest that alignment of singular vectors with features can be a sound and theoretically justified basis for feature identification in language models.
Safety-Utility Conflicts Are Not Global: Surgical Alignment via Head-Level Diagnosis
Baidu Inc.(百度公司);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);School of Computer Science, Peking University(北京大学计算机学院);State Key Laboratory of AI Safety, Institute of Computing Technology, CAS(中国科学院计算技术研究所人工智能安全重点实验室)
Temporally-Aligned Evaluation for Audio-Driven Talking Head Generation
School of Business, University of New South Wales (UNSW)(新南威尔士大学商学院);School of Engineering and Built Environment, Griffith University(格里菲斯大学工程与环境学院);Data61/CSIRO(Data61/澳大利亚国家科学委员会)
Learning aligned EEG representations with subject-specific encoders
University of São Paulo(圣保罗大学);Université Paris-Saclay, Inria TAU team, LISN-CNRS(巴黎萨克雷大学,Inria TAU团队,LISN-CNRS);Institut de neuromodulation, GHU Paris, psychiatrie et neurosciences, centre hospitalier Sainte-Anne, pôle hospitalo-universitaire 15, Université Paris Cité(神经调控研究所,GHU巴黎,精神病学与神经科学,圣安娜医院,大学医院中心15区,巴黎西岱大学);Federal University of ABC (UFABC)(ABC联邦大学);Yneuro;Swartz Center for Computational Neuroscience (SCCN), Institute for Neural Computation (INC), University of California San Diego(斯沃茨计算神经科学中心,神经计算研究所,加州大学圣地亚哥分校)
Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Models
Skolkovo Institute of Science and Technology(斯科尔科沃科学技术研究院);AI Foundation and Algorithm Lab(人工智能基金会与算法实验室);Moscow Institute of Physics and Technology(莫斯科物理技术学院);Artificial Intelligence Research Institute (AIRI)(人工智能研究院);CNRS(法国国家科学研究中心);Université Paris Cité(巴黎西岱大学)
Vision Transformer attention alignment with human visual perception in aesthetic object evaluation
Escuela de Informática y Telecomunicaciones, Universidad Diego Portáles(戴维·波特莱斯大学信息与电信学院);Department of Specific Didactics, University of Cordoba(科尔多瓦大学特定教学系);Facultad de Ingeniería y Ciencias, Universidad Adolfo Ibáñez(阿道弗·伊巴涅斯大学工程与科学学院)
本文通过眼动实验与 DINO 预训练 ViT 的注意图对比,发现特定注意头和高斯参数下可较好对齐人类审美视觉注意。
HARVE: Hacking-Aware Reward-Head Vector Editing for Robust Reward Models
Carnegie Mellon University(卡内基梅隆大学);University of Virginia(弗吉尼亚大学);Harvard University(哈佛大学);Stanford University(斯坦福大学);University of Michigan(密歇根大学);New Jersey Institute of Technology(新泽西理工学院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))