Neural FOXP2——面向大型语言模型目标语言改进的语言特定神经元引导
Neural FOXP2 -- Language Specific Neuron Steering for Targeted Language Improvement in LLMs
AI总结：
提出Neural FOXP2方法，通过定位语言神经元、计算引导方向和施加稀疏激活偏移，将模型默认语言从英语切换为印地语或西班牙语，实现可控的语言主导性。
摘要：
LLMs通过训练成为多语言模型，但其通用语言通常是英语，反映了英语在预训练中的主导地位。其他语言保留在参数记忆中，但被系统性抑制。我们认为语言默认性由稀疏、低秩的控制电路（语言神经元）支配，可以机械地隔离并安全引导。我们引入Neural FOXP2，通过引导语言特定神经元，使模型以选定语言（印地语或西班牙语）为主。Neural FOXP2分三个阶段进行：(i) 定位：我们训练每层的SAE，使每个激活分解为一小组活跃特征组件。对于每个特征，我们量化英语与印地语/西班牙语的选择性，基于整体logit质量向目标语言令牌集的提升。将排名靠前的特征追溯回其最强贡献单元，得到紧凑的语言神经元集。(ii) 引导方向：我们通过谱低秩分析定位可控的语言转换几何。对于每层，我们构建英语到目标激活差异矩阵，并执行逐层SVD以提取主导语言变化的奇异方向。特征间隙和有效秩谱识别出紧凑的引导子空间和经验选择的干预窗口（这些方向最强且最稳定）。(iii) 引导：我们对语言神经元应用有符号的稀疏激活偏移。具体地，在低到中层，我们沿目标语言主导方向添加正向引导，并对英语神经元在零空间施加补偿性负偏移，实现可控的目标语言默认性。
英文摘要：
LLMs are multilingual by training, yet their lingua franca is often English, reflecting English language dominance in pretraining. Other languages remain in parametric memory but are systematically suppressed. We argue that language defaultness is governed by a sparse, low-rank control circuit, language neurons, that can be mechanistically isolated and safely steered. We introduce Neural FOXP2, that makes a chosen language (Hindi or Spanish) primary in a model by steering language-specific neurons. Neural FOXP2 proceeds in three stages: (i) Localize: We train per-layer SAEs so each activation decomposes into a small set of active feature components. For every feature, we quantify English vs. Hindi/Spanish selectivity overall logit-mass lift toward the target-language token set. Tracing the top-ranked features back to their strongest contributing units yields a compact language-neuron set. (ii) Steering directions: We localize controllable language-shift geometry via a spectral low-rank analysis. For each layer, we build English to target activation-difference matrices and perform layerwise SVD to extract the dominant singular directions governing language change. The eigengap and effective-rank spectra identify a compact steering subspace and an empirically chosen intervention window (where these directions are strongest and most stable). (iii) Steer: We apply a signed, sparse activation shift targeted to the language neurons. Concretely, within low to mid layers we add a positive steering along the target-language dominant directions and a compensating negative shift toward the null space for the English neurons, yielding controllable target-language defaultness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
