RAIN：你的语言模型无需微调即可实现自我对齐
RAIN: Your Language Models Can Align Themselves without Finetuning
AI总结：
针对大语言模型与人类偏好不一致的问题，提出无需微调、无需对齐数据的可回退自归回推理方法RAIN，通过自我评估与回退机制实现模型自我对齐，显著提升模型无害性与真实性。
摘要：
大语言模型（LLM）常表现出与人类偏好不一致的问题。以往研究通常先收集人类偏好数据，再通过强化学习或指令微调（即微调步骤）对预训练模型进行对齐。相比之下，无需对齐数据就能让冻结的LLM完成对齐更具吸引力。本研究探索了这一设定的可行性。我们发现，通过整合自我评估与回退机制，未对齐的LLM可通过自我提升直接生成符合人类偏好的响应。我们提出了一种全新的推理方法——可回退自回归推理（Rewindable Auto-regressive INference，RAIN），让预训练LLM能够评估自身生成内容，并利用评估结果指导回退与生成，以保障AI安全。值得注意的是，RAIN无需额外数据用于模型对齐，也弃权（不执行）任何训练、梯度计算或参数更新。经GPT-4和人类评估的实验结果证明了RAIN的有效性：在HH数据集上，RAIN将LLaMA 30B的无害率从普通推理的82%提升至97%，同时保持了有用率；在TruthfulQA数据集上，RAIN将已完成良好对齐的LLaMA-2-chat 13B模型的真实性提升了5%。
英文摘要：
Large language models (LLMs) often demonstrate inconsistencies with human preferences. Previous research typically gathered human preference data and then aligned the pre-trained models using reinforcement learning or instruction tuning, a.k.a. the finetuning step. In contrast, aligning frozen LLMs without requiring alignment data is more appealing. This work explores the potential of the latter setting. We discover that by integrating self-evaluation and rewind mechanisms, unaligned LLMs can directly produce responses consistent with human preferences via self-boosting. We introduce a novel inference method, Rewindable Auto-regressive INference (RAIN), that allows pre-trained LLMs to evaluate their own generation and use the evaluation results to guide rewind and generation for AI safety. Notably, RAIN operates without the need of extra data for model alignment and abstains from any training, gradient computation, or parameter updates. Experimental results evaluated by GPT-4 and humans demonstrate the effectiveness of RAIN: on the HH dataset, RAIN improves the harmlessness rate of LLaMA 30B from 82% of vanilla inference to 97%, while maintaining the helpfulness rate. On the TruthfulQA dataset, RAIN improves the truthfulness of the already-well-aligned LLaMA-2-chat 13B model by 5%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
