MEPT：作为流形映射器的专家混合提示微调
MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper
AI总结：
本文提出MEPT框架，通过混合专家架构集成多个提示专家，自适应学习多样数据分布，在SuperGLUE上提升准确率并大幅减少激活提示。
摘要：
将深度神经网络视为流形映射器，预训练后微调范式可解释为两阶段过程：预训练建立广泛知识库，微调调整模型参数以激活特定神经通路，从而与目标流形对齐。尽管现有微调方法取得成功，但其僵化的参数空间限制了动态激活合适神经通路的能力，使其难以灵活适应多样且不断演变的数据分布。基于此观点，我们提出一种新方法——专家混合提示微调（MEPT），作为一种高效有效的流形映射框架。MEPT利用混合专家架构，通过集成多个提示专家来自适应地学习多样且非平稳的数据分布。实证评估表明，MEPT在SuperGLUE上优于多个最先进的参数高效基线，在平均准确率上取得显著提升（例如1.94%），同时将激活的提示显著减少79.25%。MEPT的有效性还得到流形学习理论见解的进一步支持，并通过神经激活通路可视化结果得到验证。
英文摘要：
Considering deep neural networks as manifold mappers, the pretrain-then-fine-tune paradigm can be interpreted as a two-stage process: pretrain establishes a broad knowledge base, and fine-tune adjusts the model parameters to activate specific neural pathways to align with the target manifold. Although prior fine-tuning approaches demonstrate success, their rigid parameter space limits their ability to dynamically activate appropriate neural pathways, rendering them ill-equipped to adapt flexibly to the diverse and evolving data distributions. In light of this view, we propose a novel approach, Mixture of Expert Prompt Tuning (MEPT), as an effective and efficient manifold-mapping framework. MEPT leverages the Mixture of Experts architecture by integrating multiple prompt experts to adaptively learn diverse and non-stationary data distributions. Empirical evaluations demonstrate that MEPT outperforms several state-of-the-art parameter efficient baselines on SuperGLUE, achieving notable improvements in mean accuracy (e.g., 1.94%) while significantly reducing activated prompts by 79.25%. The effectiveness of MEPT is further supported by theoretical insights from manifold learning and validated through neural activation pathway visualization results. Our code is avaliable at https://runjia.tech/emnlp_mept/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
