元奖励语言模型：基于LLM作为元评判者的自我改进对齐
Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
AI总结：
提出Meta-Rewarding步骤让LLM评判自身评判以提升对齐能力，实验证明该方法在无监督下显著提高了Llama-3-8B-Instruct的指令遵循与评判表现。
摘要：
大型语言模型（LLMs）在许多领域正迅速超越人类知识。虽然改进这些模型传统上依赖于昂贵的人类数据，但最近的自我奖励机制（Yuan et al., 2024）表明，LLMs可以通过评判自身的回复来改进，而不是依赖人类标注者。然而，现有方法主要集中在改进模型回复而非评判能力上，导致在迭代训练中迅速饱和。为了解决这个问题，我们在自我改进过程中引入了一个新颖的Meta-Rewarding步骤，模型在该步骤中评判自身的评判，并利用该反馈来完善其评判技能。令人惊讶的是，这种无监督方法提高了模型评判和遵循指令的能力，正如Llama-3-8B-Instruct在AlpacaEval 2上的胜率从22.9%提高到39.4%，在Arena-Hard上从20.6%提高到29.1%所证明的那样。这些结果有力地表明了无需人类监督的自我改进模型的潜力。
英文摘要：
Large Language Models (LLMs) are rapidly surpassing human knowledge in many domains. While improving these models traditionally relies on costly human data, recent self-rewarding mechanisms (Yuan et al., 2024) have shown that LLMs can improve by judging their own responses instead of relying on human labelers. However, existing methods have primarily focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training. To address this issue, we introduce a novel Meta-Rewarding step to the self-improvement process, where the model judges its own judgements and uses that feedback to refine its judgment skills. Surprisingly, this unsupervised approach improves the model's ability to judge {\em and} follow instructions, as demonstrated by a win rate improvement of Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on Arena-Hard. These results strongly suggest the potential for self-improving models without human supervision.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
