平衡增强性、无害性与通用能力：通过直接RLHF增强对话式大语言模型
Balancing Enhancement, Harmlessness, and General Capabilities: Enhancing Conversational LLMs with Direct RLHF
AI总结：
针对对话式LLM经SFT后基础能力下降、有毒输出增多的问题，提出绕过SFT直接实施无害RLHF的方法，构建的Mistral-Plus在保留通用能力的同时提升对话能力与安全性，11项任务表现优于同规模开源模型及指令版。
摘要：
在对话式大语言模型（LLM）的最新进展中，出现了一个值得关注的趋势：许多新型基础大语言模型在经过监督微调（SFT）后，其基础能力出现了知识衰减。这一过程通常会导致基础模型能力遗忘或下降等问题。此外，微调后的模型难以与用户偏好对齐，在特定提示下会不经意间增加有毒输出的生成。为应对这些挑战，我们采用了一种创新方法，完全绕过SFT，直接实施无害的基于人类反馈的强化学习（RLHF）。我们的方法不仅保留了基础模型的通用能力，还显著提升了其对话能力，同时 notably 减少了有毒输出的生成。该方法对需要细致理解和生成回复的领域（如客户服务）具有重要意义。我们将这一方法应用于最受欢迎的基础模型Mistral，由此创建了Mistral-Plus。我们在11项通用任务上的验证表明，Mistral-Plus的表现优于同等规模的开源基础模型及其对应的指令微调版本。重要的是，Mistral-Plus的对话能力得到了显著提升，这表明其在安全性和用户偏好对齐方面相较传统SFT模型取得了实质性进步。
英文摘要：
In recent advancements in Conversational Large Language Models (LLMs), a concerning trend has emerged, showing that many new base LLMs experience a knowledge reduction in their foundational capabilities following Supervised Fine-Tuning (SFT). This process often leads to issues such as forgetting or a decrease in the base model's abilities. Moreover, fine-tuned models struggle to align with user preferences, inadvertently increasing the generation of toxic outputs when specifically prompted. To overcome these challenges, we adopted an innovative approach by completely bypassing SFT and directly implementing Harmless Reinforcement Learning from Human Feedback (RLHF). Our method not only preserves the base model's general capabilities but also significantly enhances its conversational abilities, while notably reducing the generation of toxic outputs. Our approach holds significant implications for fields that demand a nuanced understanding and generation of responses, such as customer service. We applied this methodology to Mistral, the most popular base model, thereby creating Mistral-Plus. Our validation across 11 general tasks demonstrates that Mistral-Plus outperforms similarly sized open-source base models and their corresponding instruct versions. Importantly, the conversational abilities of Mistral-Plus were significantly improved, indicating a substantial advancement over traditional SFT models in both safety and user preference alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
