基于原则驱动、以最少人类监督从零开始实现语言模型自我对齐
Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
AI总结：
本文提出 SELF-ALIGN 方法，通过原则驱动推理和大语言模型生成能力，在极少人类监督下实现模型自我对齐，并基于 LLaMA-65b 开发出超越 Text-Davinci-003 和 Alpaca 的助手 Dromedary。
摘要：
近期的人工智能助手智能体（如 ChatGPT）主要依赖带有人工标注的监督微调（SFT）和基于人类反馈的强化学习（RLHF）来使大语言模型（LLMs）的输出与人类意图对齐，以确保其有用、合乎伦理且可靠。然而，这种依赖可能显著限制人工智能助手智能体的真正潜力，原因在于获取人类监督的高昂成本，以及与之相关的质量、可靠性、多样性、自一致性和不良偏见等问题。为应对这些挑战，我们提出了一种名为 SELF-ALIGN 的新方法，该方法将原则驱动的推理与大语言模型的生成能力相结合，在最少人类监督下实现人工智能智能体的自我对齐。我们的方法包含四个阶段：首先，我们使用一个大语言模型生成合成提示，并采用主题引导的方法来增强提示多样性；其次，我们使用一小套人类编写的原则供人工智能模型遵循，并通过从（原则应用的）示范中进行上下文学习来引导大语言模型，从而对用户查询生成有用、合乎伦理且可靠的回答；第三，我们使用高质量的自我对齐回答对原始大语言模型进行微调，使所得模型能够直接为每个查询生成理想回答，而不再需要原则集和示范；最后，我们提供一个细化步骤，以解决回答过于简短或间接的问题。将 SELF-ALIGN 应用于 LLaMA-65b 基础语言模型，我们开发了一个名为 Dromedary 的人工智能助手。在不到 300 行人类标注（包括少于 200 个种子提示、16 条通用原则和 5 个用于上下文学习的示例）的情况下，Dromedary 在多种设置下的基准数据集上显著超越了多个最先进人工智能系统的性能，包括 Text-Davinci-003 和 Alpaca。
英文摘要：
Recent AI-assistant agents, such as ChatGPT, predominantly rely on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback (RLHF) to align the output of large language models (LLMs) with human intentions, ensuring they are helpful, ethical, and reliable. However, this dependence can significantly constrain the true potential of AI-assistant agents due to the high cost of obtaining human supervision and the related issues on quality, reliability, diversity, self-consistency, and undesirable biases. To address these challenges, we propose a novel approach called SELF-ALIGN, which combines principle-driven reasoning and the generative power of LLMs for the self-alignment of AI agents with minimal human supervision. Our approach encompasses four stages: first, we use an LLM to generate synthetic prompts, and a topic-guided method to augment the prompt diversity; second, we use a small set of human-written principles for AI models to follow, and guide the LLM through in-context learning from demonstrations (of principles application) to produce helpful, ethical, and reliable responses to user's queries; third, we fine-tune the original LLM with the high-quality self-aligned responses so that the resulting model can generate desirable responses for each query directly without the principle set and the demonstrations anymore; and finally, we offer a refinement step to address the issues of overly-brief or indirect responses. Applying SELF-ALIGN to the LLaMA-65b base language model, we develop an AI assistant named Dromedary. With fewer than 300 lines of human annotations (including < 200 seed prompts, 16 generic principles, and 5 exemplars for in-context learning). Dromedary significantly surpasses the performance of several state-of-the-art AI systems, including Text-Davinci-003 and Alpaca, on benchmark datasets with various settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
