混合专家Transformer中任务路由的理论模型
A theoretical model for task routing in mixture-of-expert transformers
AI总结：
通过离散语言模型证明单层MoE Transformer可利用专家实现任务专业化，支持经验发现。
摘要：
混合专家（MoE）层使得在保持推理计算固定的情况下扩展Transformer模型成为可能。尽管在前沿MoE Transformer模型的实证研究中观察到了任务-专家专业化现象，但现有的理论工作使用连续混合模型进行分析，无法有效建模自然语言。一个重要的问题是使用离散语言模型从理论上解释Transformer MoE模型中的任务-专家专业化。为此，我们通过句法模板和有限键值字典表示结构化知识，并正式证明单层MoE Transformer可以通过使用专注于相应任务的专家来编码知识。我们的构造展示了查询如何被路由到唯一的、特定于任务的专家，其大小仅取决于给定任务的内在复杂度（即其句法模板和事实字典的组合大小）。我们的构造为MoE模型中局部化知识回路的实证结果提供了理论支持。我们通过实验评估模型在不同MoE损失函数下的性能来支持我们的理论发现。
英文摘要：
Mixture-of-experts (MoE) layers enable the scaling of transformer models while keeping the inference compute fixed. While task-expert specialization has been observed in empirical studies of frontier MoE transformer models, existing theoretical work analyzes this using continuous mixture models that cannot be used to model natural language effectively. An important open question is to \textit{theoretically explain task-expert specialization in transformer MoE models using discrete models of language}. To address this, we represent structured knowledge via syntactic templates and finite key-value dictionaries, and prove formally that a single-layer MoE transformer can encode knowledge by using experts that specialize in the corresponding tasks. Our construction shows how queries are routed to unique, task-specific experts whose size depends solely on the intrinsic complexity of the given task (i.e. the combined size of its syntactic templates and factual dictionary). Our construction provides a theoretical support for empirical results on localized knowledge circuits in MoE models. We support our theoretical findings with experiments evaluating model performance under varying MoE loss functions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
