arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.20179cs.CLcs.AIcs.LG
收藏

跨模型上下文神经反馈揭示大语言模型的(有限)激活控制

Cross-Model In-Context Neurofeedback Reveals (Limited) Activation Control in LLMs

MILA – The Quebec AI Institute(米拉-魁北克人工智能研究所);MATS Research(MATS研究机构);LawZero

Davide Baldelli, Robyn Kozierok, Mirko Bronzi, Damiano Fornasiere

AI总结:

该研究通过上下文神经反馈实验测试10个9B至32B的开放权重LLM,发现仅3个模型有微弱一致的激活控制,无模型能降低激活探测器分数,仅1个模型可提高,表明LLM存在有限的激活控制。

摘要:

对激活向量的探测被广泛用于检测语言模型的不良行为,但其成功依赖于模型无法控制自身激活所编码内容的假设。人类可通过神经反馈学习控制甚至单个神经元的放电,近期研究探讨这种能力是否延伸至语言模型,却得出了相互矛盾的结论。目前尚无实验设计能区分任务胜任能力与模型对自身激活的真实访问权限。我们通过上下文神经反馈会话填补这一空白,其中“主体”模型被要求最大化或最小化从其激活中导出的分数,每次交互时都会将该分数反馈给它。在首个实验中,句子由主体模型或第二个“生成器”模型生成,而分数始终来自主体模型的激活。我们测试了来自5个家族的10个开放权重模型,规模从9B到32B不等。部分模型能显著改变自身分数,但改变幅度不超过同等能力的生成器模型。这是因为该任务无需特权访问,还是因为主体模型在处理句子时控制自身激活,无论句子由谁生成?为回答此问题,我们每次交互时都预先填充固定句子作为主体模型的回复,并要求其提高或降低分数。10个模型中有3个表现出微小但一致的控制效果。我们进一步探究模型是否能控制自身激活到足以欺骗激活探测器的程度。我们让模型重复探测器会标记的句子,例如针对欺骗探测器的欺骗性回答,并要求其提高或降低探测器的分数。没有模型学会降低分数,但有1个模型能提高分数。模型对自身激活的上下文控制可能尚未对激活探测器构成威胁,但这种控制存在于少数模型中,且需随着模型的发展而被追踪。

英文摘要:

Probes on activation vectors are widely used to detect misbehavior in language models, but their success relies on the assumption that models cannot control what their activations encode. Humans can learn, through neurofeedback, to control the firing of even single neurons, and recent studies investigated whether this ability extends to language models, reaching conflicting conclusions. No experimental design so far disentangles competence at the task from genuine access to one's own activations. We fill this gap with in-context neurofeedback sessions, in which a "subject" model is asked to maximize (or minimize) a score derived from its activations, shown back to it at every turn. In our first experiment, the sentences are generated either by the subject or by a second "writer" model, while the score always derives from the activations of the subject. We test ten open-weight models from five families, with sizes from 9B to 32B. Some models move their own scores considerably, but no more than a writer of comparable competence moves them. Is this because the task requires no privileged access, or because the subject controls its activations while processing the sentences, whoever wrote them? To answer this, we prefill a fixed sentence as the subject's reply at every turn and ask it to increase or decrease the score. Three out of ten models show small but consistent control effects. We further ask whether models can control their activations to the point of fooling an activation monitor. We make models repeat sentences a monitor would flag, such as a deceptive answer for a deception monitor, and ask them to raise or lower the monitor's score. No model learns to lower it, but one model can raise it. In-context control of a model's own activations might not yet be a threat to activation monitors, but it exists in a few models and needs to be tracked as models evolve.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑