LangFIR: 从单语数据中发现稀疏语言特定特征以实现语言引导
LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering
AI总结：
LangFIR通过仅使用少量单语数据和随机令牌序列，发现稀疏语言特定特征，从而在多语言生成控制中实现高准确率，优于单语基线和依赖平行数据的方法。
摘要：
大型语言模型（LLMs）在多语言能力方面表现强劲，但可靠地控制输出语言仍具挑战性。表示层面引导通过在推理时向模型激活添加语言特定向量来解决这一问题，但识别残差流中的语言特定方向通常依赖于多语言或平行数据，这可能成本高昂。稀疏自编码器（SAEs）将残差激活分解为可解释的、稀疏的特征方向，为这种搜索提供了自然基础，但现有基于SAE的方法面临相同的数据限制。我们引入LangFIR（通过随机令牌过滤的语言特征识别），一种仅使用少量单语数据和随机令牌序列来发现语言特定SAE特征的方法。许多SAE特征被目标语言输入一致激活，但不编码语言身份。随机令牌序列凸显这些语言无关特征，使LangFIR能够过滤它们并隔离稀疏的语言特定特征。我们证明这些特征非常稀疏，对目标语言高度选择性，并且因果重要：方向消融仅增加相应语言的交叉熵损失。使用这些特征构建多语言生成控制的引导向量，LangFIR在三个模型（Gemma 3 1B、Gemma 3 4B和Llama 3.1 8B）、三个数据集和十二种目标语言上实现了最佳的平均BLEU分数，优于单语基线，甚至超越依赖平行数据的方法。我们的结果表明，在多语言LLMs中语言身份局部化于可由单语数据发现的稀疏特征方向中。代码可在https://anonymous.4open.science/r/LangFIR-C0F5/获取。
英文摘要：
Large language models (LLMs) show strong multilingual capabilities, yet reliably controlling the language of their outputs remains difficult. Representation-level steering addresses this by adding language-specific vectors to model activations at inference time, but identifying language-specific directions in the residual stream often relies on multilingual or parallel data that can be expensive to obtain. Sparse autoencoders (SAEs) decompose residual activations into interpretable, sparse feature directions and offer a natural basis for this search, yet existing SAE-based approaches face the same data constraint. We introduce LangFIR (Language Feature Identification via Random-token Filtering), a method that discovers language-specific SAE features using only a small amount of monolingual data and random-token sequences. Many SAE features consistently activated by target-language inputs do not encode language identity. Random-token sequences surface these language-agnostic features, allowing LangFIR to filter them out and isolate a sparse set of language-specific features. We show that these features are extremely sparse, highly selective for their target language, and causally important: directional ablation increases cross-entropy loss only for the corresponding language. Using these features to construct steering vectors for the multilingual generation control task, LangFIR achieves the best average accuracy x BLEU among steering methods across three models (Gemma 3 1B, Gemma 3 4B, and Llama 3.1 8B), three datasets, and twelve target languages, outperforming the strongest monolingual baseline by up to 4.7x and surpassing methods that use parallel data. Our results suggest that language identity in multilingual LLMs is localized in a sparse set of feature directions discoverable with monolingual data. Code is available at https://github.com/JaMussCraft/LangFIR
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
