THOR-MoE：面向神经机器翻译的分层任务引导与上下文响应路由
THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation
AI总结：
针对神经机器翻译中现有稀疏MoE路由存在的任务知识难获取、未考虑上下文的问题，提出分层任务引导与上下文响应的THOR-MoE，在多领域、多语言翻译基准上性能更优，且兼容现有路由方案。
摘要：
稀疏混合专家（Mixture-of-Experts, MoE）已在神经机器翻译（neural machine translation, NMT）领域取得显著进展。然而，当前MoE方案存在两个可能导致性能次优的局限：1）它们直接将NMT的任务知识（例如领域/语言学特定知识）应用于MoE，这类知识在实际应用中通常难以获取，且忽略了自然成组的领域/语言学属性；2）专家选择仅依赖局部的词元表示，未考虑上下文，而上下文能从全局视角充分掌握每个词元的状态。\n为解决上述局限，我们提出THOR-MoE，为MoE配备分层任务引导与上下文响应路由策略。具体而言，该方法1）首先预测领域/语言标签，随后提取混合的领域/语言表示，以分层方式分配任务级专家；2）注入上下文信息，从预先选定的任务级专家集合中优化词元路由，帮助每个词元被准确路由至更专业、更适配的专家。\n我们在不同架构的多领域翻译和多语言翻译基准上开展了大量实验，结果一致证明THOR-MoE性能优异。此外，THOR-MoE是即插即用模块，兼容现有的Top-$k$和Top-$p$路由方案，确保其可广泛应用于各类MoE架构。例如，与原始Top-$p$路由相比，该上下文感知方式在多领域翻译任务上，仅激活不到22%的参数，就能实现平均0.75 BLEU的提升。
英文摘要：
The sparse Mixture-of-Experts (MoE) has achieved significant progress for neural machine translation (NMT). However, there exist two limitations in current MoE solutions which may lead to sub-optimal performance: 1) they directly use the task knowledge of NMT into MoE (\emph{e.g.}, domain/linguistics-specific knowledge), which are generally unavailable at practical application and neglect the naturally grouped domain/linguistic properties; 2) the expert selection only depends on the localized token representation without considering the context, which fully grasps the state of each token in a global view. To address the above limitations, we propose THOR-MoE via arming the MoE with hierarchical task-guided and context-responsive routing policies. Specifically, it 1) firstly predicts the domain/language label and then extracts mixed domain/language representation to allocate task-level experts in a hierarchical manner; 2) injects the context information to enhance the token routing from the pre-selected task-level experts set, which can help each token to be accurately routed to more specialized and suitable experts. Extensive experiments on multi-domain translation and multilingual translation benchmarks with different architectures consistently demonstrate the superior performance of THOR-MoE. Additionally, the THOR-MoE operates as a plug-and-play module compatible with existing Top-$k$~\cite{shazeer2017} and Top-$p$~\cite{huang-etal-2024-harder} routing schemes, ensuring broad applicability across diverse MoE architectures. For instance, compared with vanilla Top-$p$~\cite{huang-etal-2024-harder} routing, the context-aware manner can achieve an average improvement of 0.75 BLEU with less than 22\% activated parameters on multi-domain translation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
