arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.01945cs.CL
收藏

LFQA-E: 仔细评估长形式问答评测

LFQA-E: Carefully Benchmarking Long-form QA Evaluation

Shanghai Jiao Tong University(上海交通大学);Shanghai AI Lab(上海人工智能实验室);Tsimnghua University(塔密纳大学);Beijing University of Posts and Telecommunications(北京邮电大学);Zhejiang University(浙江大学);University of Tokyo(东京大学)

Yuchen Fan, Chen Lin, Xin Zhong, Shuo Zhang, Heng Zhou, Yuchen Zhang, Mingyu Liang, Chengxing Xie, Ermo Hua, Gang Chen, Zhizhou He, Cheng Huang, Ning Ding, Bowen Zhou

AI总结:

LFQA-E是一个多语言、基于参考的长形式问答评测基准,用于严格评估自动指标的性能,揭示现有指标在捕捉长回答信息方面的不足。

摘要:

长形式问答(LFQA)涉及生成对开放性问题的全面、段落级回答,这在评估上带来了重大挑战,因为信息丰富性和灵活的响应格式使得评估变得困难。现有的LFQA评测基准往往缺乏参考答案,并且在规模和主题覆盖上有限,降低了其可靠性。为了解决这一差距,我们引入了LFQA-E,一个精心构建、多语言且基于参考的基准,旨在严格评估自动指标的LFQA能力。LFQA-E包含1618个问题和7323对比较,涵盖15个主题,来源于多样化的来源,如在线查询和考试问题,从而能够全面评估评估指标。我们使用LFQA-E检验了五类指标,涵盖17种具体方法。结果表明,现有的自动指标在与人类判断相比时表现不佳,突显了它们无法捕捉长形式回答中密集信息的能力。此外,我们还详细分析了这些指标的失败案例和泛化能力,为未来LFQA评估方法的发展提供了见解。该基准和代码可在https://github.com/YuchenFan48/LFQA-E上获得。

英文摘要:

Long-Form Question Answering (LFQA) involves generating comprehensive, paragraph-level responses to open-ended questions, which poses a significant challenge for evaluation due to the richness of information and flexible response format. Existing LFQA-evaluation benchmarks often lack reference answers and are limited in size and topic coverage, reducing their reliability. To address this gap, we introduce LFQA-E, a well-constructed, multilingual, and reference-based benchmark designed to rigorously evaluate automatic metrics for LFQA. LFQA-E comprises 1618 questions and 7323 pairwise comparisons across 15 topics, drawn from diverse sources such as online queries and examination questions, thereby enabling a comprehensive assessment of evaluation metrics. We examine five categories of metrics, encompassing 17 specific methods, using LFQA-E. The results demonstrate that none of the existing automatic metrics perform comparably to human judgments, highlighting their inability to capture the dense information in long-form responses. Furthermore, we present a detailed analysis of the failure cases and the generalization capacity of these metrics, offering insights to guide the future development of LFQA evaluation methods. The benchmark and code are available at https://github.com/YuchenFan48/LFQA-E.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑