CORTEX：在稠密语言模型中学习共享与特化
CORTEX: Learning to Share and Specialize in Dense Language Models
- Simon Fraser University（西蒙弗雷泽大学）
- Southern University of Science and Technology（南方科技大学）
- The University of British Columbia（不列颠哥伦比亚大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
CORTEX提出一种受学习动态启发的框架，在稠密语言模型中通过参数分组和梯度相似性学习内部模块化，实现共享与特化，并在多个规模模型上取得最优性能。
AI中文摘要：
大型语言模型在异构数据混合上进行训练，其中不同的知识领域既需要共享知识，也需要特化知识。现有的模块化方法通常强加显式组件，或在训练后通过可解释性分析发现模块。在这项工作中，我们提出了CORTEX，一个受学习动态启发的框架，用于在稠密语言模型内部学习模块化。CORTEX将可训练矩阵划分为参数组，并从领域条件的梯度和跨领域梯度相似性中学习模块分配。我们引入了选择性损伤分数和模块-领域互信息来表征目标领域的损伤效应和一致性，并分析了模块分配如何影响分配偏差与更新幅度之间的权衡。使用160M、Qwen3-8B和Qwen3-32B骨干模型的实验表明，CORTEX在合成领域精确匹配上达到最高，平均困惑度降低最大，同时在真实领域评估中保持竞争力并形成可识别的模块。
英文摘要：
Large language models are trained on heterogeneous data mixtures, where different knowledge domains require both shared knowledge and specialization. Existing modular approaches typically impose explicit components or discover modules through interpretability analysis after training. In this work, we propose CORTEX, a learning dynamics-inspired framework that learns internal modularization within dense language models. CORTEX partitions trainable matrices into parameter groups and learns module assignments from domain-conditioned gradient and cross-domain gradient similarity. We introduce the selective lesion score and module-domain mutual information to characterize the target-domain lesion effects and alignment, and analyze how module assignment affects the trade-off between assignment bias and update magnitude. Experiments with 160M, Qwen3-8B, and Qwen3-32B backbone models show that CORTEX achieves the highest synthetic-domain exact match and largest average perplexity reduction, while remaining competitive on real-domain evaluations and forming identifiable modules.
