SenseJudge: 以人为中心的偏好驱动判断框架
SenseJudge: Human-Centric Preference-Driven Judgment Framework
AI总结：
提出SenseJudge框架和SenseBench基准，通过融入用户偏好实现个性化判断和模型排名，实验证明其优于现有方法。
摘要：
大型语言模型（LLMs）作为判断器在评估模型响应等各种场景中正成为一种日益被接受的范式。然而，现有的判断方法通常依赖于使用固定偏好数据训练的评判器，这往往忽视了多样化的用户偏好，难以适应真实的人机对话场景。为了解决这些局限性，我们提出了SenseJudge，一个由人类偏好驱动的可定制判断框架，以及SenseBench，一个源自真实世界多轮交互的多样且具有挑战性的指令遵循基准。我们将自动判断框架和基准应用于两个任务：（1）LLMs作为个性化判断器，以及（2）模型排名。我们进行了大量实验，结果表明SenseJudge框架在LLMs作为个性化判断器任务中超越了其他判断方法和模型，并实现了与真实人类感知一致的模型排名。此外，我们对位置偏差和一致性进行了分析，并进行了消融研究，证实了SenseJudge的鲁棒性。
英文摘要：
Large Language Models (LLMs) as judges across various scenarios such as assessing model responses is becoming an increasingly accepted paradigm. However, existing judgment approaches often rely on trained judgers using fixed preference data, which tend to overlook diverse user preferences and struggle to adapt to real-world human-AI dialogue scenarios. To address these limitations, we propose SenseJudge, a customizable judgment framework driven by human preferences and SenseBench, a diverse and challenging instruction-following benchmark derived from real-world multi-turn interactions. We applied the automatic judgment framework and benchmark to two tasks: (1) LLMs as personalized judges, and (2) model ranking. We conducted extensive experiments, and the results demonstrate that the SenseJudge framework surpasses other judgment methods and models in the LLMs-as-personalized-judges task and achieves model ranking that aligns with real human sense. Additionally, we conducted analyses on position bias and consistency, alongside ablation studies, which affirmed the robustness of SenseJudge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
