分布式特化：大型语言模型中的稀有词元神经元
Distributed Specialization: Rare-Token Neurons in Large Language Models
AI总结：
本研究揭示大型语言模型通过分布式特化机制处理稀有词元，发现平台神经元等三区域影响力层级，并证明该机制无需专用路由电路，为模型编辑和效率优化提供新见解。
摘要：
大型语言模型（LLMs）在表示和生成稀有词元方面存在困难，尽管这些词元在专业领域中非常重要。我们研究了LLMs是否通过离散模块化架构或分布式参数级分化来发展内部特化机制。通过对多个模型家族的最终层MLP神经元进行系统分析，我们发现稀有词元处理通过“分布式特化”出现：功能协调但空间分布的亚网络，展现出三种不同的组织原则。首先，我们识别出一个可复现的三区域影响力层级，包括高影响力平台神经元（也称为稀有词元神经元）、幂律衰减神经元和最小贡献神经元，这种层级在常见词元处理中不存在。其次，平台神经元表现出协调的激活模式（有效维度降低），同时保持空间分布而非形成离散簇。第三，这些特化机制通过标准注意力路径普遍可访问，无需专用路由电路。训练动态揭示，功能特化通过参数分化逐渐出现，特化神经元发展出日益重尾的权重相关谱，与重尾自正则化特征一致。我们的发现表明，LLMs通过共享架构内的分布式协调而非专家混合式模块化来处理稀有词元。这些结果为可解释模型编辑、计算效率优化以及理解Transformer网络中涌现的功能组织提供了见解。
英文摘要：
Large language models (LLMs) struggle with representing and generating rare tokens despite their importance in specialized domains. We investigate whether LLMs develop internal specialization mechanisms through discrete modular architectures or distributed parameter-level differentiation. Through systematic analysis of final-layer MLP neurons across multiple model families, we discover that rare-token processing emerges via \textit{distributed specialization}: functionally coordinated but spatially distributed subnetworks that exhibit three distinct organizational principles. First, we identify a reproducible three-regime influence hierarchy comprising highly influential plateau neurons(also termed as rare-token neurons), power-law decay neurons, and minimally contributing neurons, which is absent in common-token processing. Second, plateau neurons demonstrate coordinated activation patterns (reduced effective dimensionality) while remaining spatially distributed rather than forming discrete clusters. Third, these specialized mechanisms are universally accessible through standard attention pathways without requiring dedicated routing circuits. Training dynamics reveal that functional specialization emerges gradually through parameter differentiation, with specialized neurons developing increasingly heavy-tailed weight correlation spectra consistent with Heavy-Tailed Self-Regularization signatures. Our findings establish that LLMs process rare-tokens through distributed coordination within shared architectures rather than mixture-of-experts-style modularity. These results provide insights for interpretable model editing, computational efficiency optimization, and understanding emergent functional organization in transformer networks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
