局部化LLM中的渐进式局部化
Progressive Localisation in Localist LLMs
AI总结:
本文提出了一种渐进式局部化方法,通过自适应语义块划分和陡峭多项式调度,在保持性能的同时提升LLM的可解释性。
摘要:
本文证明了渐进式局部化,即从早期的分布式层到后期的局部化层逐步增加注意力局部性,是创建可解释的大型语言模型(LLMs)的最佳架构,同时保持性能。通过系统地在《人工智能超智能心理学》上微调GPT-2,我们评估了五种局部性配置:两种均匀基线(完全分布式和完全局部化)以及三种渐进的多项式调度。我们研究了在网络深度中战略性应用时,可解释性约束是否可以与自然语义结构对齐。我们证明了渐进式语义局部化,结合自适应语义块划分与陡峭的多项式局部性调度,能够在接近基线语言建模性能的同时提供可解释的注意力模式。多个独立的训练运行使用不同的随机种子证明,结果在统计上是稳健且高度可重复的。该方法显著优于固定窗口局部化和朴素的均匀局部性约束。分析表明,通过低保真约束保持灵活性可以保留模型容量并提供可解释性益处,而陡峭的调度在决策关键的最终层集中局部性,同时在早期层保持分布式学习,从而实现接近基线的注意力分布特征。这些发现表明,可解释性机制应与语义结构对齐,以实现实际的性能-可解释性权衡,为可信的AI系统服务。
英文摘要:
This paper demonstrates that progressive localization, the gradual increase of attention locality from early distributed layers to late localized layers, represents the optimal architecture for creating interpretable large language models (LLMs) while preserving performance. Through systematic experimentation with GPT-2 fine-tuned on The Psychology of Artificial Superintelligence, we evaluate five locality configurations: two uniform baselines (fully distributed and fully localist) and three progressive polynomial schedules. We investigate whether interpretability constraints can be aligned with natural semantic structure while being applied strategically across network depth. We demonstrate that progressive semantic localization, combining adaptive semantic block partitioning with steep polynomial locality schedules, achieves near-baseline language modeling performance while providing interpretable attention patterns. Multiple independent training runs with different random seeds establish that results are statistically robust and highly reproducible. The approach dramatically outperforms both fixed-window localization and naive uniform locality constraints. Analysis reveals that maintaining flexibility through low-fidelity constraints preserves model capacity while providing interpretability benefits, and that steep schedules concentrating locality in decision-critical final layers while preserving distributed learning in early layers achieve near-baseline attention distribution characteristics. These findings demonstrate that interpretability mechanisms should align with semantic structure to achieve practical performance-interpretability tradeoffs for trustworthy AI systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。