MultEval: 支持LLM-as-a-Judge评估标准的协作对齐
MultEval: Supporting Collaborative Alignment for LLM-as-a-Judge Evaluation Criteria
AI总结：
研究探讨了LLM-as-a-Judge评估标准的协作制定过程，提出MultEval系统支持多方协商和迭代修订标准，提升评估透明度与一致性。
摘要：
LLM-as-a-judge方法已成为评估模型行为的可扩展解决方案，但现有工具难以支持多方协作定义评估标准。本文通过形成性研究揭示了协作制定标准的挑战，包括建立共享理解、对齐不同专家的价值观及将细微判断转化为可执行的评估标准。基于这些发现，我们开发了MultEval系统，通过共识理论支持多方诊断分歧、迭代修订标准并保持透明度，展示了协作共识形成对标准演化的影响力。
英文摘要：
LLM-as-a-judge approaches have emerged as a scalable solution for evaluating model behaviors, yet they rely on evaluation criteria often created by a single individual, embedding that person's assumptions, priorities, and interpretive lens. In practice, defining such criteria is a collaborative and contested process involving multiple stakeholders with different values, interpretations, and priorities; an aspect largely unsupported by existing tools. To examine this problem in depth, we present a formative study examining how stakeholders collaboratively create, negotiate, and refine evaluation criteria for LLM-as-a-judge systems. Our findings reveal challenges in human oversight, including difficulties in establishing shared understanding, aligning values across stakeholders with different expertise and priorities, and translating nuanced human judgments into criteria that are interpretable and actionable for LLM judges. Based on these insights, we developed MultEval, a system that supports collaborative criteria by enabling multiple evaluators to surface and diagnose disagreements using consensus-building theory, iteratively revise criteria with attached examples and proposal history, and maintain transparency over how judgments are encoded into an automated evaluator. We further report a case study in which a team of domain experts used MultEval to collaboratively author criteria, illustrating how coordination and collaborative consensus-making shape criteria evolution.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
