PMoE：面向持续学习的渐进式专家混合与非对称Transformer
PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learning
AI总结：
本文提出PMoE，通过非对称Transformer中浅层保留通用知识、深层逐步增加专家并由基于深层特征的路由器分配新知识，有效缓解LLM持续学习中的灾难性遗忘，并在多个数据集上超越现有方法。
摘要：
大型语言模型（LLMs）在持续学习中面临灾难性遗忘的重大挑战，即新信息会覆盖先前获得的知识。这一局限导致了显著的环境和经济浪费。在本研究中，我们提出了PMoE，即渐进式专家混合与非对称Transformer，旨在通过采用非对称设计来最小化遗忘，其中浅层专门用于通用知识，深层用于新知识。PMoE在深层中引入逐步增加的专家，并采用一个路由器将新知识高效地分配给合适的专家。该路由器位于深层附近，利用聚合了整合信息的深层特征。这使得路由器能够高效运行，将新知识分配给逐步增加的深层专家。在TRACE数据集和通用语言理解数据集上的大量实验表明，所提出的PMoE优于先前的最先进方法。
英文摘要：
Large Language Models (LLMs) encounter significant challenges in continual learning due to catastrophic forgetting, where new information overwrites previously acquired knowledge. This limitation leads to substantial environmental and economic waste. In this study, we introduce the PMoE, Progressive Mixture of Experts with Asymmetric Transformer, which aims to minimize forgetting by utilizing an asymmetric design with shallow layers dedicated to general knowledge and deep layers for new knowledge. PMoE incorporates progressively added experts in deep layers and a router that allocates new knowledge to the appropriate experts efficiently. The router, positioned adjacent to the deep layers, utilizes deep features aggregating consolidated information. This enables the router to perform efficiently, allocating new knowledge to the appropriate experts, which progressively increase in the deep layers. Extensive experiments on TRACE datasets and general language understanding datasets demonstrate that the proposed PMoE outperforms previous state-of-the-art approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
