迈向使语言模型与文本反馈对齐
Towards Aligning Language Models with Textual Feedback
AI总结：
提出仅依赖语言建模的 ALT 方法，通过条件化文本反馈对齐语言模型，在毒性降低上优于 PPO，并以 20% 样本达到摘要任务同等性能。
摘要：
我们提出了 ALT（ALignment with Textual feedback，文本反馈对齐），这是一种使语言模型与以文本表达的用户偏好保持一致的方法。我们认为，文本具有更强的表达能力，使用户能够提供比简单比较偏好更丰富的反馈，而这种更丰富的反馈可以带来更高效、更有效的对齐。ALT 通过让模型生成以文本反馈为条件来对齐模型。我们的方法仅依赖语言建模技术，并且只需极少的超参数调优；尽管如此，它仍具备基于 RL 的对齐算法的主要优势，并能够有效地从文本反馈中学习。我们在降低毒性、摘要生成和对话回复生成等不同任务上探究了文本反馈的有效性与效率。我们发现，ALT 在降低毒性任务上优于 PPO，同时在摘要生成任务上仅使用 20% 的样本就能达到与其相当的性能。我们还探究了如何将 ALT 与现有 LLM 提供的反馈结合使用，其中考察了 LLM 提供受约束和不受约束的文本反馈的情形。我们还概述了使模型与自然语言反馈对齐的未来方向。
英文摘要：
We present ALT (ALignment with Textual feedback), an approach that aligns language models with user preferences expressed in text. We argue that text offers greater expressiveness, enabling users to provide richer feedback than simple comparative preferences and this richer feedback can lead to more efficient and effective alignment. ALT aligns the model by conditioning its generation on the textual feedback. Our method relies solely on language modeling techniques and requires minimal hyper-parameter tuning, though it still presents the main benefits of RL-based alignment algorithms and can effectively learn from textual feedback. We explore the efficacy and efficiency of textual feedback across different tasks such as toxicity reduction, summarization, and dialog response generation. We find that ALT outperforms PPO for the task of toxicity reduction while being able to match its performance on summarization with only 20% of the samples. We also explore how ALT can be used with feedback provided by an existing LLM where we explore an LLM providing constrained and unconstrained textual feedback. We also outline future directions to align models with natural language feedback.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
