RLVF：从口头反馈中学习而不过度泛化
RLVF: Learning from Verbal Feedback without Overgeneralization
AI总结：
针对大型语言模型仅通过提示引入口头反馈易导致行为过度泛化的问题，提出了C3PO方法，通过生成合成偏好数据集进行约束微调，在有效应用反馈的同时最小化对无关上下文的影响，将过度泛化降低30%。
摘要：
大型语言模型（LLMs）部署的上下文多样性，要求其具备修改或定制默认模型行为的能力，以纳入细微的要求和偏好。指定此类模型调整的便捷接口是高级口头反馈，例如“在给我的老板起草电子邮件时不要使用表情符号”。然而，尽管编写高级反馈远比收集用于人类反馈强化学习（RLHF）的标注简单，我们发现，仅用此类反馈提示模型会导致反馈过度泛化到与其不相关的上下文中。我们研究了在不产生此类过度泛化的情况下纳入口头反馈的问题，这启发了一种新方法：带约束偏好优化的上下文批评（C3PO）。C3PO利用一段高级反馈生成一个小型合成偏好数据集，具体说明应（和不应）如何应用该反馈。然后，它根据合成偏好数据对模型进行微调，同时在反馈不适用的提示上最小化与原始模型的散度。我们的实验结果表明，我们的方法能有效地将口头反馈应用于相关场景，同时保留其他上下文的现有行为。对于人类和GPT-4生成的高级反馈，C3PO有效地遵循了给定的反馈，其效果与上下文基线相当，同时将过度泛化减少了30%。
英文摘要：
The diversity of contexts in which large language models (LLMs) are deployed requires the ability to modify or customize default model behaviors to incorporate nuanced requirements and preferences. A convenient interface to specify such model adjustments is high-level verbal feedback, such as "Don't use emojis when drafting emails to my boss." However, while writing high-level feedback is far simpler than collecting annotations for reinforcement learning from human feedback (RLHF), we find that simply prompting a model with such feedback leads to overgeneralization of the feedback to contexts where it is not relevant. We study the problem of incorporating verbal feedback without such overgeneralization, inspiring a new method Contextualized Critiques with Constrained Preference Optimization (C3PO). C3PO uses a piece of high-level feedback to generate a small synthetic preference dataset specifying how the feedback should (and should not) be applied. It then fine-tunes the model in accordance with the synthetic preference data while minimizing the divergence from the original model for prompts where the feedback does not apply. Our experimental results indicate that our approach effectively applies verbal feedback to relevant scenarios while preserving existing behaviors for other contexts. For both human- and GPT-4-generated high-level feedback, C3PO effectively adheres to the given feedback comparably to in-context baselines while reducing overgeneralization by 30%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
