MoxE：面向高效语言建模的熵感知路由的xLSTM专家混合模型
MoxE: Mixture of xLSTM Experts with Entropy-Aware Routing for Efficient Language Modeling
AI总结：
本文提出MoxE架构，将xLSTM与MoE框架结合，并引入熵感知路由机制动态分配token给专家，同时采用辅助损失函数，以提升大语言模型的可扩展性和效率。
摘要：
本文介绍了MoxE，一种新颖的架构，它将扩展长短期记忆网络（xLSTM）与专家混合（MoE）框架协同结合，以应对大语言模型（LLMs）中关键的可扩展性和效率挑战。所提出的方法有效利用了xLSTM创新的记忆结构，同时通过MoE策略性地引入稀疏性，从而大幅降低计算开销。我们方法的核心是一种新颖的基于熵的路由机制，旨在动态地将token路由到专门的专家，从而确保高效且均衡的资源利用。这种熵感知能力使该架构能够有效处理稀有token和常见token，其中mLSTM模块被优先用于处理稀有token。为了进一步增强泛化能力，我们引入了一组辅助损失函数，包括基于熵的损失和分组均衡损失，以确保稳健的性能和高效的训练。理论分析和实证评估严格证明，与现有方法相比，MoxE实现了显著的效率提升和增强的有效性，标志着可扩展LLM架构的显著进步。
英文摘要：
This paper introduces MoxE, a novel architecture that synergistically combines the Extended Long Short-Term Memory (xLSTM) with the Mixture of Experts (MoE) framework to address critical scalability and efficiency challenges in large language models (LLMs). The proposed method effectively leverages xLSTM's innovative memory structures while strategically introducing sparsity through MoE to substantially reduce computational overhead. At the heart of our approach is a novel entropy-based routing mechanism, designed to dynamically route tokens to specialized experts, thereby ensuring efficient and balanced resource utilization. This entropy awareness enables the architecture to effectively manage both rare and common tokens, with mLSTM blocks being favored to handle rare tokens. To further enhance generalization, we introduce a suite of auxiliary losses, including entropy-based and group-wise balancing losses, ensuring robust performance and efficient training. Theoretical analysis and empirical evaluations rigorously demonstrate that MoxE achieves significant efficiency gains and enhanced effectiveness compared to existing approaches, marking a notable advancement in scalable LLM architectures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
