CompassJudger-1：一体化裁判模型助力模型评估与进化
CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution
AI总结：
本文提出首个开源一体化裁判大语言模型CompassJudger-1，兼具评分、比较、批评和通用任务能力，并配套发布主观评估基准JudgerBench，以统一标准推动LLM评估方法学发展。
摘要：
高效且准确的评估对于大语言模型（LLMs）的持续改进至关重要。在各种评估方法中，主观评估因其与真实使用场景和人类偏好的高度一致性而受到广泛关注。然而，基于人工的评估成本高昂且缺乏可复现性，这使得精确的自动化评估器（裁判模型）在这一过程中变得不可或缺。在本报告中，我们介绍了CompassJudger-1，这是首个开源的“一体化”裁判大语言模型。CompassJudger-1是一个通用型大语言模型，展现出卓越的多功能性。它能够：1. 作为奖励模型执行单一评分和双模型比较；2. 按照指定格式进行评估；3. 生成批评意见；4. 像通用大语言模型一样执行多样化任务。为了在统一设置下评估不同裁判模型的评估能力，我们还建立了JudgerBench，这是一个涵盖各种主观评估任务并覆盖广泛主题的新基准。CompassJudger-1为各种评估任务提供了全面的解决方案，同时保持了适应多样化需求的灵活性。CompassJudger和JudgerBench均已发布并可供研究社区使用，地址为https://github.com/open-compass/CompassJudger。我们相信，通过开源这些工具，可以促进合作并加速大语言模型评估方法学的进步。
英文摘要：
Efficient and accurate evaluation is crucial for the continuous improvement of large language models (LLMs). Among various assessment methods, subjective evaluation has garnered significant attention due to its superior alignment with real-world usage scenarios and human preferences. However, human-based evaluations are costly and lack reproducibility, making precise automated evaluators (judgers) vital in this process. In this report, we introduce \textbf{CompassJudger-1}, the first open-source \textbf{all-in-one} judge LLM. CompassJudger-1 is a general-purpose LLM that demonstrates remarkable versatility. It is capable of: 1. Performing unitary scoring and two-model comparisons as a reward model; 2. Conducting evaluations according to specified formats; 3. Generating critiques; 4. Executing diverse tasks like a general LLM. To assess the evaluation capabilities of different judge models under a unified setting, we have also established \textbf{JudgerBench}, a new benchmark that encompasses various subjective evaluation tasks and covers a wide range of topics. CompassJudger-1 offers a comprehensive solution for various evaluation tasks while maintaining the flexibility to adapt to diverse requirements. Both CompassJudger and JudgerBench are released and available to the research community athttps://github.com/open-compass/CompassJudger. We believe that by open-sourcing these tools, we can foster collaboration and accelerate progress in LLM evaluation methodologies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
