在弱监督下保持真实性：利用同伴预测评估和训练大语言模型
Truthfulness Despite Weak Supervision: Evaluating and Training LLMs Using Peer Prediction
AI总结：
通过同伴预测方法，利用弱监督提升大语言模型的真实性，有效对抗欺骗并增强评估可靠性。
摘要：
大型语言模型（LLMs）的评估和训练依赖于监督，但对困难任务的强监督往往不可用，尤其是在评估前沿模型时。在这种情况下，模型被证明会利用基于这种不完美监督的评估，导致欺骗性结果。然而，在LLM研究中，大量关于机制设计的研究专注于博弈论激励相容性，即利用弱监督获取诚实且有信息的回答。基于这一文献，我们引入了同伴预测方法用于模型评估和训练。该方法通过基于相互可预测性的度量标准，奖励诚实且有信息的回答，而非欺骗性和无信息的回答，且不需真实标签。我们展示了该方法的有效性和对欺骗的抵抗力，通过理论保证和在参数多达405B的模型上的实证验证。我们证明，使用基于同伴预测的奖励训练一个8B模型，可以恢复由于先前恶意微调导致的真实性下降，即使奖励由未经过微调的0.135B语言模型生成。在评估方面，与需要强且可信的法官的LLM-as-a-Judge不同，我们发现同伴预测中存在反比例性质，即令人惊讶的是，随着专家与参与者之间能力差距的扩大，对欺骗的抵抗力增强，使得能够利用弱监督可靠地评估强大的模型。特别是，当面对比法官大5-20倍的欺骗模型时，LLM-as-a-Judge会比随机猜测更差，而同伴预测在这样的差距较大时表现优异，包括在超过100倍大小差异的情况下。
英文摘要：
The evaluation and post-training of large language models (LLMs) rely on supervision, but strong supervision for difficult tasks is often unavailable, especially when evaluating frontier models. In such cases, models are demonstrated to exploit evaluations built on such imperfect supervision, leading to deceptive results. However, underutilized in LLM research, a wealth of mechanism design research focuses on game-theoretic incentive compatibility, i.e., eliciting honest and informative answers with weak supervision. Drawing from this literature, we introduce the peer prediction method for model evaluation and post-training. It rewards honest and informative answers over deceptive and uninformative ones, using a metric based on mutual predictability and without requiring ground truth labels. We demonstrate the method's effectiveness and resistance to deception, with both theoretical guarantees and empirical validation on models with up to 405B parameters. We show that training an 8B model with peer prediction-based reward recovers most of the drop in truthfulness due to prior malicious finetuning, even when the reward is produced by a 0.135B language model with no finetuning. On the evaluation front, in contrast to LLM-as-a-Judge which requires strong and trusted judges, we discover an inverse scaling property in peer prediction, where, surprisingly, resistance to deception is strengthened as the capability gap between the experts and participants widens, enabling reliable evaluation of strong models with weak supervision. In particular, LLM-as-a-Judge become worse than random guess when facing deceptive models 5-20x the judge's size, while peer prediction thrives when such gaps are large, including in cases with over 100x size difference.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
