arXiv 2405.005572025-06-01cs.CLcs.AI

Mixture of insighTful Experts (MoTE)：思维链与专家混合在自我对齐中的协同作用

Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment

The Hong Kong University of Science and Technology（香港科学与技术大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Southern University of Science and Technology（南方科技大学）

Zhili Liu, Yunhao Gou, Kai Chen, Lanqing Hong, Jiahui Gao, Fei Mi, Yu Zhang, Zhenguo Li, Xin Jiang, Qun Liu, James T. Kwok

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对LLMs自我对齐挑战，提出MoTE框架，通过结合四阶段结构化推理链与步骤级路由的多LoRA专家混合架构，显著提升模型安全性、抗越狱与过度拒绝能力，性能媲美OpenAI o1模型。

摘要：

随着大型语言模型（LLMs）的能力不断增强，将这些模型与人类价值观对齐仍是一项重大挑战。近期研究表明，推理能力对模型安全性有显著贡献，而集成混合专家架构可进一步增强对齐效果。在本研究中，我们探讨一个基础问题：如何将推理能力与MoE架构有效融入LLMs的自我对齐过程？我们提出了Mixture of insighTful Experts (MoTE)，这是一个协同结合推理链与专家混合以改进自我对齐的新颖框架。从数据视角来看，MoTE采用包含四个关键阶段的结构化推理链：问题分析、回答指导、安全回答和安全检查。该方法通过多步推理增强安全性，即使对于较小且能力较弱的LLMs（例如7B模型）也证明有效。从架构视角来看，MoTE采用带有步骤级路由的多LoRA框架，其中每个专家专门负责特定的推理步骤。这种设计消除了对平衡损失的需求，确保了训练稳定性，并支持自适应推理长度。实验结果表明，MoTE显著提升了模型安全性、越狱抵抗力和过度拒绝能力，实现了与OpenAI最先进的o1模型相当的性能。

英文摘要：

As the capabilities of large language models (LLMs) continue to expand, aligning these models with human values remains a significant challenge. Recent studies show that reasoning abilities contribute significantly to model safety, while integrating Mixture-of-Experts (MoE) architectures can further enhance alignment. In this work, we address a fundamental question: How to effectively incorporate reasoning abilities and MoE architectures into self-alignment process in LLMs? We propose Mixture of insighTful Experts (MoTE), a novel framework that synergistically combines reasoning chains and expert mixtures to improve self-alignments. From a data perspective, MoTE employs a structured reasoning chain comprising four key stages: Question Analysis, Answer Guidance, Safe Answer, and Safety Checking. This approach enhances safety through multi-step reasoning and proves effective even for smaller and less powerful LLMs (e.g., 7B models). From an architectural perspective, MoTE adopts a multi-LoRA framework with step-level routing, where each expert is dedicated to a specific reasoning step. This design eliminates the need for balance losses, ensures stable training, and supports adaptive inference lengths. Experimental results demonstrate that MoTE significantly improves model safety, jailbreak resistance, and over-refusal capabilities, achieving performance comparable to OpenAI's state-of-the-art o1 model.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。