T-REX：具有语义感知直觉的秩一专家混合用于多任务大语言模型微调
T-REX: Mixture-of-Rank-One-Experts with Semantic-aware Intuition for Multi-task Large Language Model Finetuning
AI总结：
提出T-REX框架，利用秩一专家混合构建LoRA权重，以线性参数开销二次扩展专家子空间，并通过语义聚类隐式指导路由，实现高效多任务微调，在14个数据集上以更少参数提升准确率。
摘要：
大语言模型（LLMs）在多样化的多任务微调中面临显著的适配挑战。混合专家（MoE）通过动态架构提供了一种有前景的解决方案，能够实现有效的任务解耦。然而，增加MoE专家的数量会带来大量的参数和计算开销，并且由于朴素的路由机制，性能提升有限。在本文中，我们设计了一个新颖的框架，即秩一专家混合（T-REX），它利用超低秩专家的组合来在预训练LLMs上构建LoRA权重。秩一专家支持一种混合匹配机制，以线性参数开销将专家的向量子空间二次扩展，以最优效率实现近似误差减少。此外，T-REX为路由器提供隐式指导，利用训练嵌入的固有语义聚类作为先验知识，实现跨专家的优化特征分配，以获得更平滑的收敛。广泛的理论和实证结果表明，T-REX在多样任务中实现了卓越的效率和泛化能力。与其他基于LoRA的方法相比，T-REX在14个公共数据集上以约30%-40%更少的可训练参数实现了高达1.78%的平均准确率提升。代码可在https://github.com/RoyZry98/T-REX-Pytorch获取。
英文摘要：
Large language models (LLMs) encounter significant adaptation challenges in diverse multitask finetuning. Mixture-of-experts (MoE) provides a promising solution with a dynamic architecture, enabling effective task decoupling. However, scaling up the number of MoE experts incurs substantial parameter and computational overheads and suffers from limited performance gain due to naive routing mechanisms. In this paper, we design a novel framework, mix\underline{\textbf{T}}ure\underline{\textbf{-}}of-\underline{\textbf{R}}ank-on\underline{\textbf{E}}-e\underline{\textbf{X}}perts (\texttt{T-REX}), which leverages the combination of ultra-low rank experts to construct LoRA weights on pretrained LLMs. The rank-1 experts enable a mix-and-match mechanism to quadratically expand the vector subspace of experts with linear parameter overheads, achieving approximate error reduction with optimal efficiency. In addition, T-REX offers implicit guidance to the router, leveraging the inherent semantic clustering of training embeddings as prior knowledge, enabling optimized feature allocation across experts for a smoother convergence. Extensive theoretical and empirical results demonstrate that T-REX achieves superior efficiency and generalizability across diverse tasks. Compared with other LoRA-based methods, T-REX achieves up to 1.78\% mean accuracy improvement with around 30\%-40\% less trainable parameters across 14 public datasets. \href{https://github.com/RoyZry98/T-REX-Pytorch}{Code} is available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
