LINKAGE：基于LLM的非事实型问答评估中面向多质量参考的列表式排序方法
LINKAGE: Listwise Ranking among Varied-Quality References for Non-Factoid QA Evaluation via LLMs
AI总结：
本文提出LINKAGE方法，利用LLM对候选答案在按质量排序的参考列表中进行列表式排序，以改进非事实型问答评估，并在三个数据集上验证其与人工标注相关性显著优于逐点和成对方法。
摘要：
非事实型（NF）问答（QA）由于潜在答案多样且缺乏客观标准，评估具有挑战性。常用的自动评估指标如ROUGE或BERTScore无法准确衡量语义相似性或来自不同视角的答案。近年来，大语言模型（LLMs）因其在各类NLP任务上的出色表现而被用于NFQA评估。常见方法包括对每个候选答案进行逐点评分以及在答案之间进行成对比较。受学习排序方法从逐点、成对到列表式演进的启发，我们提出了一种新颖的列表式NFQA评估方法，该方法利用LLMs对候选答案在一组按质量降序排列的参考答案列表中进行排序。此外，对于没有多等级或任何黄金答案的NF问题，我们利用LLMs生成不同质量的参考答案列表，以支持列表式评估。在三个NFQA数据集（即ANTIQUE、TREC-DL-NF和WebGLM）上的大量实验结果表明，与自动评分以及常见的逐点和成对方法相比，我们的方法与人工标注具有显著更高的相关性。
英文摘要：
Non-Factoid (NF) Question Answering (QA) is challenging to evaluate due to diverse potential answers and no objective criterion. The commonly used automatic evaluation metrics like ROUGE or BERTScore cannot accurately measure semantic similarities or answers from different perspectives. Recently, Large Language Models (LLMs) have been resorted to for NFQA evaluation due to their compelling performance on various NLP tasks. Common approaches include pointwise scoring of each candidate answer and pairwise comparisons between answers. Inspired by the evolution from pointwise to pairwise to listwise in learning-to-rank methods, we propose a novel listwise NFQA evaluation approach, that utilizes LLMs to rank candidate answers in a list of reference answers sorted by descending quality. Moreover, for NF questions that do not have multi-grade or any golden answers, we leverage LLMs to generate the reference answer list of various quality to facilitate the listwise evaluation. Extensive experimental results on three NFQA datasets, i.e., ANTIQUE, the TREC-DL-NF, and WebGLM show that our method has significantly higher correlations with human annotations compared to automatic scores and common pointwise and pairwise approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
