Probes on activation vectors are widely used to detect misbehavior in language models, but their success relies on the assumption that models cannot control what their activations encode. Humans can learn, through neurofeedback, to control the firing of even single neurons, and recent studies investigated whether this ability extends to language models, reaching conflicting conclusions. No experimental design so far disentangles competence at the task from genuine access to one's own activations. We fill this gap with in-context neurofeedback sessions, in which a "subject" model is asked to maximize (or minimize) a score derived from its activations, shown back to it at every turn. In our first experiment, the sentences are generated either by the subject or by a second "writer" model, while the score always derives from the activations of the subject. We test ten open-weight models from five families, with sizes from 9B to 32B. Some models move their own scores considerably, but no more than a writer of comparable competence moves them. Is this because the task requires no privileged access, or because the subject controls its activations while processing the sentences, whoever wrote them? To answer this, we prefill a fixed sentence as the subject's reply at every turn and ask it to increase or decrease the score. Three out of ten models show small but consistent control effects. We further ask whether models can control their activations to the point of fooling an activation monitor. We make models repeat sentences a monitor would flag, such as a deceptive answer for a deception monitor, and ask them to raise or lower the monitor's score. No model learns to lower it, but one model can raise it. In-context control of a model's own activations might not yet be a threat to activation monitors, but it exists in a few models and needs to be tracked as models evolve.
Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations
Neurosciences Graduate Program University of California San Diego(加州大学圣地亚哥分校神经科学研究生项目);School of Psychology Georgia Tech(佐治亚理工学院心理学系);Department of Psychology New York University(纽约大学心理学系);Department of Neurobiology University of California San Diego(加州大学圣地亚哥分校神经生物学系)
ReLU$^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs
Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Renmin University of China(中国人民大学)
LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations
Oxford Internet Institute, University of Oxford(牛津大学牛津互联网研究所);FLAIR, University of Oxford(牛津大学FLAIR);Department of Computer Science, University College London(伦敦大学学院计算机科学系)
Achieving binary weight and activation for LLMs using Post-Training Quantization
MAIS, Institute of Automation, Chinese Academy of Sciences(自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Central Media Technology Institute, Huawei Technologies Ltd.(华为技术有限公司媒体技术研究所)
Internal Value Alignment in Large Language Models through Controlled Value Vector Activation
University of Science and Technology of China(中国科学技术大学);State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室);Gaoling School of Artificial Intelligence(光明人工智能学院);Renmin University of China(中国人民大学);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大型模型与智能治理重点实验室);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(下一代智能搜索与推荐工程技术研究中心);Tsinghua University(清华大学);Huawei Technologies Co. Ltd(华为技术有限公司)
ZorBA: Zeroth-order Federated Fine-tuning of LLMs with Heterogeneous Block Activation
Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, Canada(电气与计算机工程系,不列颠哥伦比亚大学);Department of Computer Science and Engineering, Southern University of Science and Technology, China(计算机科学与工程系,南方科技大学)
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University(应用信息与应用数学博士学院,奥布达大学);John von Neumann Faculty of Informatics, Obuda University(约翰·冯·诺伊曼信息学院,奥布达大学)
Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons
Department of Computer Science and Technology, BNRist(计算机科学与技术系,BNRist);Shenzhen International Graduate School(深圳国际研究生院);KIRC, Institute for Artificial Intelligence, Tsinghua University(人工智能研究院,清华大学)
Unraveling Babel: Exploring Multilingual Activation Patterns of LLMs and Their Applications
Liangzhu Laboratory, Zhejiang University(浙江大学良渚实验室);AI Research Center, WeDoctor Cloud(微医云AI研究中心);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);School of Public Health, Zhejiang University(浙江大学公共卫生学院)
Detecting Hidden Behaviors in LLMs via Activation-matched Finetuning
Bonn-Aachen International Center for Information Technology, University of Bonn(波恩大学波恩-亚琛国际信息技术中心);Lamarr Institute for Machine Learning and Artificial Intelligence(拉马尔机器学习与人工智能研究院)
ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
Zhejiang University(浙江大学);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Innovation Institute(上海创新研究院);Fudan University(复旦大学);Chinese University of Hong Kong(中国香港大学)
Activation-Informed Merging of Large Language Models
Massachusetts Institute of Technology(麻省理工学院);Stony Brook University(石溪大学);MIT-IBM Watson AI Lab & Red Hat AI Innovation(MIT-IBM Watson AI实验室及Red Hat AI创新)