JudgeBoard: 对小语言模型进行推理评估的基准测试与增强
JudgeBoard: Benchmarking and Enhancing Small Language Models for Reasoning Evaluation
AI总结：
JudgeBoard通过多代理框架提升小语言模型的推理判断能力，展示其在数学和常识推理任务中与大模型相当的性能。
摘要：
尽管小语言模型（SLMs）在各种推理任务上显示出潜力，但其在判断答案正确性方面的能力仍不清楚，与大型语言模型（LLMs）相比。先前基于LLM作为判断框架的工作通常依赖于将候选答案与地面真实标签或其他候选答案进行比较，使用预定义的度量标准如蕴含性。然而，这种方法本质上是间接的，难以完全自动化，对推理输出的细粒度和可扩展性评估支持有限。在本工作中，我们提出了JudgeBoard，一种新颖的评估流程，直接查询模型以评估候选答案的正确性，而无需额外的答案比较。我们专注于两个核心推理领域：数学推理和科学常识推理，并使用五种基准数据集中的准确率排名和基于Elo的评分系统构建任务特定的评估排行榜，使模型能够作为裁判而非比较者进行一致的比较。为了提高轻量模型的判断性能，我们提出了MAJ（多代理判断），一种新的多代理评估框架，利用具有不同推理特征的多个相互作用的SLMs，通过协作辩论来近似LLM级别的判断准确性。实验结果揭示了在孤立判断任务中SLMs与LLMs之间存在显著的性能差距。然而，我们的MAJ框架显著提高了SLMs的可靠性和一致性。在MATH数据集中，使用较小模型作为骨干的MAJ表现相当或甚至优于其较大模型的对应物。我们的发现表明，多代理SLM系统可能在判断任务中与LLM性能相匹配或超越，对可扩展和高效的评估具有启示意义。
英文摘要：
While small language models (SLMs) have shown promise on various reasoning tasks, their ability to judge the correctness of answers remains unclear compared to large language models (LLMs). Prior work on LLM-as-a-judge frameworks typically relies on comparing candidate answers against ground-truth labels or other candidate answers using predefined metrics like entailment. However, this approach is inherently indirect and difficult to fully automate, offering limited support for fine-grained and scalable evaluation of reasoning outputs. In this work, we propose JudgeBoard, a novel evaluation pipeline that directly queries models to assess the correctness of candidate answers without requiring extra answer comparisons. We focus on two core reasoning domains: mathematical reasoning and science/commonsense reasoning, and construct task-specific evaluation leaderboards using both accuracy-based ranking and an Elo-based rating system across five benchmark datasets, enabling consistent model comparison as judges rather than comparators. To improve judgment performance in lightweight models, we propose MAJ (Multi-Agent Judging), a novel multi-agent evaluation framework that leverages multiple interacting SLMs with distinct reasoning profiles to approximate LLM-level judgment accuracy through collaborative deliberation. Experimental results reveal a significant performance gap between SLMs and LLMs in isolated judging tasks. However, our MAJ framework substantially improves the reliability and consistency of SLMs. On the MATH dataset, MAJ using smaller-sized models as backbones performs comparatively well or even better than their larger-sized counterparts. Our findings highlight that multi-agent SLM systems can potentially match or exceed LLM performance in judgment tasks, with implications for scalable and efficient assessment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
