用直接原则反馈抑制粉红大象
Suppressing Pink Elephants with Direct Principle Feedback
AI总结:
本文提出粉红大象问题,要求LLM避免讨论特定实体。通过简化Constitutional AI提出直接原则反馈(DPF)方法,在合成数据集上微调13B LLaMA 2模型,使其在推理时可控,性能显著优于基线且媲美GPT-4。
摘要:
现有的控制语言模型的方法,如RLHF和Constitutional AI,涉及确定哪些LLM行为是理想的,并将其训练到语言模型中。然而,在许多情况下,理想的是LLM在推理时是可控的,以便它们可以在具有不同需求的多种环境中使用。我们用粉红大象问题来说明这一点:指示LLM避免讨论某个实体(“粉红大象”),而是讨论一个首选实体(“灰大象”)。我们应用了Constitutional AI的一种新颖简化方法,即直接原则反馈(Direct Principle Feedback, DPF),它跳过了响应的排名,并直接在批评和修订上使用DPO。我们的结果显示,在我们的合成粉红大象数据集上进行DPF微调后,我们13B微调的LLaMA 2模型显著优于Llama-2-13B-Chat和提示基线,并且在评估粉红大象问题的精选测试集上表现与GPT-4一样好。
英文摘要:
Existing methods for controlling language models, such as RLHF and Constitutional AI, involve determining which LLM behaviors are desirable and training them into a language model. However, in many cases, it is desirable for LLMs to be controllable at inference time, so that they can be used in multiple contexts with diverse needs. We illustrate this with the Pink Elephant Problem: instructing an LLM to avoid discussing a certain entity (a ``Pink Elephant''), and instead discuss a preferred entity (``Grey Elephant''). We apply a novel simplification of Constitutional AI, Direct Principle Feedback, which skips the ranking of responses and uses DPO directly on critiques and revisions. Our results show that after DPF fine-tuning on our synthetic Pink Elephants dataset, our 13B fine-tuned LLaMA 2 model significantly outperforms Llama-2-13B-Chat and a prompted baseline, and performs as well as GPT-4 in on our curated test set assessing the Pink Elephant Problem.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。