arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.09235cs.CLcs.AI
收藏

参考引导判决:大语言模型作为法官在自由形式问答自动评估中的应用

Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QA

Faculty of Computer Science(计算机科学学院);Dalhousie University(达尔豪斯大学)

Sher Badshah, Hassan Sajjad

AI总结:

针对大型语言模型生成的开放式问答输出难以用传统指标评估的问题,提出参考引导判决方法,利用多个大语言模型作为法官,实验表明该方法在自由形式问答任务中提升了评估可靠性与准确性,且与人类评估强相关。

摘要:

大型语言模型(LLMs)作为能生成类人对话的聊天助手的出现,加剧了对可靠评估方法的需求,尤其是在开放式任务中。传统指标如EM和F1虽有用,但不足以捕捉这类生成式输出的完整语义和上下文深度。我们提出一种参考引导判决方法,通过利用多个LLMs作为法官来自动化评估过程。通过在自由形式问答任务上的实验,我们证明结合多个模型能提高评估的可靠性和准确性,尤其是在单个模型可能难以处理的任务中。结果表明其与人类评估有强相关性,确立了该方法作为传统指标可靠替代方案的地位。

英文摘要:

The emergence of Large Language Models (LLMs) as chat assistants capable of generating human-like conversations has amplified the need for robust evaluation methods, particularly for open-ended tasks. Conventional metrics such as EM and F1, while useful, are inadequate for capturing the full semantics and contextual depth of such generative outputs. We propose a reference-guided verdict method that automates the evaluation process by leveraging multiple LLMs as judges. Through experiments on free-form question-answering tasks, we demonstrate that combining multiple models improves the reliability and accuracy of evaluations, especially in tasks where a single model may struggle. The results indicate a strong correlation with human evaluations, establishing the proposed method as a reliable alternative to traditional metrics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑