LLM-as-a-Judge中教师偏好偏差的助手引导缓解方法
Assistant-Guided Mitigation of Teacher Preference Bias in LLM-as-a-Judge
AI总结：
本文针对LLM-as-a-Judge中代理评判模型学习教师模型偏好偏差的问题，提出引入无偏助手模型的三阶段去偏框架AGDe-Judge，在六个评估基准上验证了其有效降低偏差并保持性能的能力。
摘要：
LLM-as-a-Judge利用GPT-4等大语言模型（LLMs）评估LLM生成回复的质量，因其成本效益高且与人类评估高度一致而日益流行。然而，使用强大教师模型生成的评估数据来训练代理评判模型会引入一个关键但此前被忽视的问题：教师偏好偏差，即代理评判模型学会了对教师模型生成的回复产生有偏的偏好。为解决这一问题，我们提出了一种新设定，引入一个对教师模型回复不存在偏差的额外助手模型来补充训练数据。基于该设定，我们提出了AGDe-Judge，一个三阶段框架，旨在从训练数据中的标签和反馈两方面进行去偏。大量实验表明，AGDe-Judge在六个评估基准上保持强劲性能的同时，有效降低了教师偏好偏差。代码可在https://github.com/Liuz233/AGDe-Judge获取。
英文摘要：
LLM-as-a-Judge employs large language models (LLMs), such as GPT-4, to evaluate the quality of LLM-generated responses, gaining popularity for its cost-effectiveness and strong alignment with human evaluations. However, training proxy judge models using evaluation data generated by powerful teacher models introduces a critical yet previously overlooked issue: teacher preference bias, where the proxy judge model learns a biased preference for responses from the teacher model. To tackle this problem, we propose a novel setting that incorporates an additional assistant model, which is not biased toward the teacher model's responses, to complement the training data. Building on this setup, we introduce AGDe-Judge, a three-stage framework designed to debias from both the labels and feedbacks in the training data. Extensive experiments demonstrate that AGDe-Judge effectively reduces teacher preference bias while maintaining strong performance across six evaluation benchmarks. Code is available at https://github.com/Liuz233/AGDe-Judge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
