arXiv 2501.078902026-01-03cs.CLcs.AI

GRAPHMOE: 通过引入自我反思机制提升混合专家网络的认知深度

GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanism

Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Institute for Advanced Algorithms Research, Shanghai（上海先进算法研究所）；The University of Manchester（曼彻斯特大学）；The University of Pittsburgh（匹兹堡大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Sydney（悉尼大学）

Bo Lv, Chen Tang, Zifan Zheng, Bohao Yang, Kun Zhao, Ning Liao, Xiaoxing Wang, Feiyu Xiong, Zhiyu Li, Nayu Liu, Jingchi Jiang

备注：: 10 pages

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

GRAPHMOE通过引入自我反思机制，提升混合专家网络的认知深度，实现语言模型的先进性能。

摘要：

传统混合专家（MoE）网络通过使用多个较小的专家模型而不是单一大型网络而受益。然而，这些专家通常独立运作，留下了一个问题：是否通过连接这些模型可以提高MoE网络的性能。为此，我们引入GRAPHMOE，一种新的方法，旨在通过基于伪GraphMoE网络的自我反思机制来增强语言模型的认知深度。GRAPHMOE采用递归路由策略来模拟迭代思考步骤，从而促进专家节点之间的信息流动。我们使用低秩适应技术（LoRA）实现GRAPHMOE架构，并在各种基准数据集上进行了广泛实验。实验结果表明，GRAPHMOE优于其他基于LoRA的模型，实现了最先进的（SOTA）性能。此外，本研究探索了一种新的递归路由策略，可能激发进一步提升语言模型推理能力的进展。

英文摘要：

Traditional Mixture-of-Experts (MoE) networks benefit from utilizing multiple smaller expert models as opposed to a single large network. However, these experts typically operate independently, leaving a question open about whether interconnecting these models could enhance the performance of MoE networks. In response, we introduce GRAPHMOE, a novel method aimed at augmenting the cognitive depth of language models via a self-rethinking mechanism constructed on Pseudo GraphMoE networks. GRAPHMOE employs a recurrent routing strategy to simulate iterative thinking steps, thereby facilitating the flow of information among expert nodes. We implement the GRAPHMOE architecture using Low-Rank Adaptation techniques (LoRA) and conduct extensive experiments on various benchmark datasets. The experimental results reveal that GRAPHMOE outperforms other LoRA based models, achieving state-of-the-art (SOTA) performance. Additionally, this study explores a novel recurrent routing strategy that may inspire further advancements in enhancing the reasoning capabilities of language models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。