arXiv 2407.034792024-07-03cs.HC

面向LLM作为评判者的人本设计建议

Human-Centered Design Recommendations for LLM-as-a-Judge

Qian Pan, Zahra Ashktorab, Michael Desmond, Martin Santillan Cooper, James Johnson, Rahul Nair, Elizabeth Daly, Werner Geyer

备注：: 14 pages, 6 figures, Accepted for publication in ACL 2024 Workshop HuCLLM

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文通过EvaluLLM用户研究和八位专家访谈，提出设计建议以优化人类辅助的LLM-as-a-judge系统，在信任与成本之间取得平衡。

摘要：

传统的基于参考的指标，如BLEU和ROUGE，在评估大型语言模型（LLMs）生成的高度创造性或高质量文本时效果较差，或者在参考输出不可用的情况下也难以适用。虽然人工评估仍然是一种选择，但其成本高昂且难以扩展。近期使用LLMs作为评估者（LLM-as-a-judge）的工作很有前景，但信任和可靠性仍然是一个重要问题。整合人类输入对于确保用于评估的标准与人类意图一致，以及评估的稳健性和一致性至关重要。本文介绍了一项名为EvaluLLM的设计探索的用户研究，该研究使用户能够将LLMs作为可定制的评判者，促进人类参与，以在信任和节省成本的潜力之间取得平衡并保持谨慎。通过对八位领域专家的访谈，我们确定了在制定有效评估标准方面需要辅助，以使LLM-as-a-judge与实践者的偏好和期望保持一致。我们提供了发现和设计建议，以优化人类辅助的LLM-as-a-judge系统。

英文摘要：

Traditional reference-based metrics, such as BLEU and ROUGE, are less effective for assessing outputs from Large Language Models (LLMs) that produce highly creative or superior-quality text, or in situations where reference outputs are unavailable. While human evaluation remains an option, it is costly and difficult to scale. Recent work using LLMs as evaluators (LLM-as-a-judge) is promising, but trust and reliability remain a significant concern. Integrating human input is crucial to ensure criteria used to evaluate are aligned with the human's intent, and evaluations are robust and consistent. This paper presents a user study of a design exploration called EvaluLLM, that enables users to leverage LLMs as customizable judges, promoting human involvement to balance trust and cost-saving potential with caution. Through interviews with eight domain experts, we identified the need for assistance in developing effective evaluation criteria aligning the LLM-as-a-judge with practitioners' preferences and expectations. We offer findings and design recommendations to optimize human-assisted LLM-as-judge systems.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文