通过Transformer潜在子空间激活引导概念偏差
Steering Conceptual Bias via Transformer Latent-Subspace Activation
AI总结：
本研究提出梯度精炼的自适应激活引导框架（G-ACT），通过聚类激活差异并训练逐层探针，有效引导LLM科学代码生成偏向特定语言，显著提升分类准确率，实现可扩展、可解释的概念级控制。
摘要：
本研究探讨了在语言模型（LLMs）中激活潜在子空间是否能够将科学代码生成引导至特定编程语言。首先，对五个因果语言模型在科学编程提示上进行了评估，以量化它们在四种编程语言之间的基线偏差。一种静态神经元归因方法，即扰动C++或CPP令牌的最高激活MLP权重，被证明是脆弱的，并且在提示风格和模型规模上表现出有限的泛化能力。为了解决这些局限性，开发了一种梯度精炼的自适应激活引导框架（G-ACT）：将每个提示的激活差异聚类为一小组引导方向，并在线训练和精炼轻量级逐层探针，以选择适当的引导向量。在LLaMA-3.2 3B中，该方法通过将平均探针分类准确率提高15%，并且与标准ACT框架相比，早期层（0-6）将探针分类准确率提高了61.5%，可靠地将生成偏向于CPP语言。对于LLaMA-3.3 70B，其中注意力头信号变得更加分散，在关键层的定向注入仍然改善了语言选择。尽管逐层探针引入了适度的推理开销，但通过仅引导一部分层，它仍然实用，并实现了可复现的模型行为。这些结果展示了一种可扩展、可解释且高效的概念级控制机制，适用于实际的智能体系统。
英文摘要：
This work examines whether activating latent subspaces in language models (LLMs) can steer scientific code generation toward a specific programming language. Five causal LLMs were first evaluated on scientific coding prompts to quantify their baseline bias among four programming languages. A static neuron-attribution method, perturbing the highest activated MLP weight for a C++ or CPP token, proved brittle and exhibited limited generalization across prompt styles and model scales. To address these limitations, a gradient-refined adaptive activation steering framework (G-ACT) was developed: per-prompt activation differences are clustered into a small set of steering directions, and lightweight per-layer probes are trained and refined online to select the appropriate steering vector. In LLaMA-3.2 3B, this approach reliably biases generation towards the CPP language by increasing the average probe classification accuracy by 15% and the early layers (0-6) improving the probe classification accuracy by 61.5% compared to the standard ACT framework. For LLaMA-3.3 70B, where attention-head signals become more diffuse, targeted injections at key layers still improve language selection. Although per-layer probing introduces a modest inference overhead, it remains practical by steering only a subset of layers and enables reproducible model behavior. These results demonstrate a scalable, interpretable and efficient mechanism for concept-level control for practical agentic systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
