无需人工指令的大语言模型自对齐与有限样本
Human-Instruction-Free LLM Self-Alignment with Limited Samples
AI总结：
本文提出一种无需人工指令和标注奖励的 LLM 自对齐算法，通过检索高质量样本生成更多数据并迭代微调，在有限样本下实现安全、真实和指令遵循等领域的对齐。
摘要：
将大语言模型（LLMs）与人类价值观对齐是 LLM 从业者的一项重要任务。当前的对齐技术存在若干局限：（1）需要大量标注数据；（2）需要大量人工参与；（3）缺乏持续改进的系统化机制。本文研究在有限样本（例如少于 100 个）条件下将 LLM 对齐到新领域的问题。我们提出一种算法，能够在没有主动人工参与的情况下迭代地自对齐 LLM。与现有工作不同，该算法既不依赖人工编写的指令，也不依赖带标签的奖励，从而显著减少人工参与。此外，该算法能够持续自我改进对齐效果。其核心思想是：首先检索与目标领域相关的高质量样本，并将其作为上下文学习示例来生成更多样本；然后利用自生成样本迭代微调 LLM。我们表明，该方法能够释放 LLM 的自我泛化能力，在接近零人工监督的情况下完成对齐。我们在安全性、真实性和指令遵循三个基准上测试了该算法，并在对齐性能、领域适应性和可扩展性方面表现出良好效果。
英文摘要：
Aligning large language models (LLMs) with human values is a vital task for LLM practitioners. Current alignment techniques have several limitations: (1) requiring a large amount of annotated data; (2) demanding heavy human involvement; (3) lacking a systematic mechanism to continuously improve. In this work, we study aligning LLMs to a new domain with limited samples (e.g. < 100). We propose an algorithm that can self-align LLMs iteratively without active human involvement. Unlike existing works, our algorithm relies on neither human-crafted instructions nor labeled rewards, significantly reducing human involvement. In addition, our algorithm can self-improve the alignment continuously. The key idea is to first retrieve high-quality samples related to the target domain and use them as In-context Learning examples to generate more samples. Then we use the self-generated samples to finetune the LLM iteratively. We show that our method can unlock the LLMs' self-generalization ability to perform alignment with near-zero human supervision. We test our algorithm on three benchmarks in safety, truthfulness, and instruction-following, and show good performance in alignment, domain adaptability, and scalability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
