通过激活几何进行无监督特征挖掘
Unsupervised Features Mining via Activation Geometry
AI总结：
介绍通过激活几何挖掘（MAG）的无监督框架，在输入前加指令Q，用m(Q∣p)-m(p)衡量其对模型内部表征的影响，探索多种MAG，所提取特征可预测模型理解与判断，还用于选择最佳训练数据集。
摘要：
可解释性方法旨在揭示大语言模型中的特征。许多现有方法从可能反映人类偏见的人类定义概念的标记示例开始。我们引入通过激活几何挖掘（MAG），一个简单的无监督框架，通过在每个输入前加上相同的自然语言指令Q来从模型激活中提取推理特征，测量指令如何改变模型内部表征，探索八个不同的MAG。提取的推理特征预测模型自身的世界理解和判断，可近似为单个激活方向，通过激活控制改变大语言模型的决策。最后，我们使用相同方法为提示注入分类器探针选择最佳训练数据集：虽然普通激活之间的相似性与下游性能几乎无关，但基于RFD的相似性达到了94.7%的Top-1和100%的Top-2准确率。
英文摘要：
Interpretability methods aim to reveal the features represented inside large language models (LLMs). Many existing methods begin with labeled examples of a human-defined concept that may reflect human biases, and then identify how that concept is represented within the model, for example in its activation space or through other decomposition methods. We introduce \emph{Mining via Activation Geometry} (MAG), a simple unsupervised framework for extracting reasoning features from model activations by prepending the same natural-language instruction $Q$ to every input $p$, where $Q$ defines the reasoning feature of interest, such as ``Can this object be found in the desert?'' or ``Is this prompt malicious?'' We measure how the instruction changes the model's internal representation using $m(Q \mid p) - m(p)$ at a single readout point. We explore eight different MAGs. The extracted reasoning features predict the models' own world understanding and judgment, can be approximated into a single activation direction, we found that some features are more linearly represented and some less, this linear representation, which is vector steering, can change the LLMs' decisions through activation steering by injecting reasoning features. Finally, we use the same method to select the best training datasets for prompt-injection classifier probes: while similarity between ordinary activations is almost unrelated to downstream performance, RFD-based similarity achieves $94.7\%$ Top-1 and $100\%$ Top-2 accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
