不同程度的错误：仅使用错误答案对齐LLM
Varying Shades of Wrong: Aligning LLMs with Wrong Answers Only
AI总结：
针对缺乏可靠标注的挑战性任务，本文提出利用“错对错”偏好对齐LLM，通过自洽性、token概率和LLM评判等方法引导偏好并微调模型，实验证明该方法能提升模型表现并改善校准。
摘要：
在缺乏针对挑战性任务和场景的充足可靠标注时，我们如何利用潜在的错误答案拓展LLM的能力边界？我们聚焦两个研究问题：(1) LLM能否在错误选项间生成可靠的偏好？若能，(2) 基于这种“错对错”偏好的对齐是否有帮助？我们采用基于自洽性、token概率和LLM作为评判者的方法来引导出错对错偏好，并利用偏好优化方法微调语言模型。在7个LLM和8个数据集上的大量实验表明：(1) LLM确实具备区分不同程度错误的初步能力，性能比随机猜测最高提升20.9%；(2) 错对错偏好对齐有助于LLM产生较少错误甚至完全正确的答案，同时整体上改善模型校准。
英文摘要：
In the absence of abundant reliable annotations for challenging tasks and contexts, how can we expand the frontier of LLM capabilities with potentially wrong answers? We focus on two research questions: (1) Can LLMs generate reliable preferences among wrong options? And if so, (2) Would alignment with such wrong-over-wrong preferences be helpful? We employ methods based on self-consistency, token probabilities, and LLM-as-a-judge to elicit wrong-over-wrong preferences, and fine-tune language models with preference optimization approaches using these synthesized preferences. Extensive experiments with seven LLMs and eight datasets demonstrate that (1) LLMs do have preliminary capability in distinguishing various shades of wrong, achieving up to 20.9% higher performance than random guess; (2) Alignment with wrong-over-wrong preferences helps LLMs to produce less wrong and sometimes even outright correct answers, while overall improving model calibration.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
