通过稀疏特征引导实现多语言Transformer中的因果语言控制
Causal Language Control in Multilingual Transformers via Sparse Feature Steering
AI总结：
本研究利用稀疏自编码器特征在推理时引导多语言模型生成目标语言，通过修改单层单个特征实现高达90%的语言转换成功率，并保持语义一致性，展示了轻量级可解释的多语言控制方法。
摘要：
确定性控制大型多语言语言模型（LLMs）的目标生成语言仍然是一个基本挑战，特别是在零样本设置中，既没有显式语言提示，也没有微调可用。在本工作中，我们研究了稀疏自编码器（SAE）特征——先前已显示与可解释模型行为相关——是否可以在推理期间用于引导LLMs的生成语言。利用在Gemma-2B和Gemma-9B的残差流上预训练的SAE，我们识别出在英语和四种目标语言（中文、日语、西班牙语和法语）之间激活差异最显著的特征。通过仅修改一个Transformer层中的单个SAE特征，我们实现了受控的语言转换，成功率高达90%（根据FastText语言分类测量），同时根据LaBSE（语言无关的BERT句子嵌入）相似性保持语义保真度。我们的分析揭示，语言引导在Transformer的中后层最有效，并且被与语言敏感SAE特征不成比例相关的特定注意力头放大。这些结果证明了稀疏特征引导作为一种轻量级且可解释的机制，用于可控多语言生成的潜力。
英文摘要：
Deterministically controlling the target generation language of large multilingual language models (LLMs) remains a fundamental challenge, particularly in zero-shot settings where neither explicit language prompts nor fine-tuning are available. In this work, we investigate whether sparse autoencoder (SAE) features, previously shown to correlate with interpretable model behaviors, can be leveraged to steer the generated language of LLMs during inference. Leveraging pretrained SAEs on the residual streams of Gemma-2B and Gemma-9B, we identify features whose activations differ most significantly between English and four target languages: Chinese, Japanese, Spanish, and French. By modifying just a single SAE feature at one transformer layer, we achieve controlled language shifts with up to 90\% success, as measured by FastText language classification, while preserving semantic fidelity according to LaBSE (Language-Agnostic BERT Sentence Embedding) similarity. Our analysis reveals that language steering is most effective in mid-to-late transformer layers and is amplified by specific attention heads disproportionately associated with language-sensitive SAE features. These results demonstrate the promise of sparse feature steering as a lightweight and interpretable mechanism for controllable multilingual generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
