arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.11353cs.LGcs.CL
收藏

MoE-RBench:构建具有稀疏混合专家架构的可靠语言模型

$\texttt{MoE-RBench}$: Towards Building Reliable Language Models with Sparse Mixture-of-Experts

Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);The University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);MIT(麻省理工学院);Harvard University(哈佛大学);The Chinese University of Hong Kong(香港中文大学)

Guanjie Chen, Xinyu Zhao, Tianlong Chen, Yu Cheng

AI总结:

本文提出 MoE-RBench,首次从安全性、对抗鲁棒性和分布外鲁棒性三方面系统评估稀疏混合专家模型,发现通过优化训练设置可使其比稠密大语言模型更可靠。

摘要:

混合专家(MoE)作为一种有前景的框架,在扩展大语言模型(LLMs)方面越来越受欢迎。然而,MoE 的可靠性评估落后于其迅猛增长的应用。此外,当迁移到新领域时,例如在微调中,MoE 模型有时表现不如其稠密对应模型。受这一研究空白和反直觉现象的启发,我们提出了 MoE-RBench,这是首个从三个方面对稀疏混合专家(SMoE)可靠性进行的全面评估:(i)安全性与幻觉,(ii)对对抗攻击的鲁棒性,以及(iii)分布外鲁棒性。我们测试了大量模型和数据集,从这些可靠性维度将 MoE 与稠密网络进行比较。我们的实证观察表明,通过适当的超参数、训练配方和推理技术,我们可以构建比稠密 LLM 更可靠的 MoE 模型。特别是,我们发现 SMoE 的鲁棒性对基本训练设置很敏感。我们希望这项研究能为如何将预训练的 MoE 模型适配到其他任务提供更深入的见解,以实现更高的生成安全性、质量和稳定性。代码可在 https://github.com/UNITES-Lab/MoE-RBench 获取。

英文摘要:

Mixture-of-Experts (MoE) has gained increasing popularity as a promising framework for scaling up large language models (LLMs). However, the reliability assessment of MoE lags behind its surging applications. Moreover, when transferred to new domains such as in fine-tuning MoE models sometimes underperform their dense counterparts. Motivated by the research gap and counter-intuitive phenomenon, we propose $\texttt{MoE-RBench}$, the first comprehensive assessment of SMoE reliability from three aspects: $\textit{(i)}$ safety and hallucination, $\textit{(ii)}$ resilience to adversarial attacks, and $\textit{(iii)}$ out-of-distribution robustness. Extensive models and datasets are tested to compare the MoE to dense networks from these reliability dimensions. Our empirical observations suggest that with appropriate hyperparameters, training recipes, and inference techniques, we can build the MoE model more reliably than the dense LLM. In particular, we find that the robustness of SMoE is sensitive to the basic training settings. We hope that this study can provide deeper insights into how to adapt the pre-trained MoE model to other tasks with higher-generation security, quality, and stability. Codes are available at https://github.com/UNITES-Lab/MoE-RBench

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑