PEDANTS:廉价但有效且可解释的答案等价性评估
PEDANTS: Cheap but Effective and Interpretable Answer Equivalence
AI总结:
针对现有问答评估指标在处理LLMs冗长答案时面临的成本高、数据多样性不足及依赖性强等问题,本文提供了源自Trivia的评分标准与数据集,并提出一种比精确匹配和BERTScore更稳定、高效且可解释的答案等价性评估方法PEDANTS。
摘要:
问答(QA)要取得进展,必须能判断答案是否正确,但当前的答案正确性(AC)指标在处理大型语言模型(LLMs)生成的冗长自由文本答案时存在困难。当前短文本问答评估面临两个挑战:缺乏多样化的评估数据风格,以及过度依赖昂贵且缓慢的LLMs。基于LLM的评分器与人类评分相关性更好,但这项昂贵任务仅在有限的QA数据集上进行了测试。我们通过提供源自Trivia社区、用于评估机器问答的评分标准与数据集来解决这些问题。我们还提出了一种高效且可解释的QA评估方法,比精确匹配和神经方法(BERTScore)更稳定。
英文摘要:
Question answering (QA) can only make progress if we know if an answer is correct, but current answer correctness (AC) metrics struggle with verbose, free-form answers from large language models (LLMs). There are two challenges with current short-form QA evaluations: a lack of diverse styles of evaluation data and an over-reliance on expensive and slow LLMs. LLM-based scorers correlate better with humans, but this expensive task has only been tested on limited QA datasets. We rectify these issues by providing rubrics and datasets for evaluating machine QA adopted from the Trivia community. We also propose an efficient, and interpretable QA evaluation that is more stable than an exact match and neural methods(BERTScore).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。