arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.05707cs.CLcs.AI
收藏

QGEval:面向问题生成的多维度评估基准测试

QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation

Shaanxi Province Key Laboratory of Big Data Knowledge Engineering(陕西省大数据知识工程重点实验室)

Weiping Fu, Bifan Wei, Jianxiang Hu, Zhongmin Cai, Jun Liu

AI总结:

提出多维度评估基准QGEval,从7个维度评估问题生成模型与自动指标,发现现有模型在可回答性上表现不佳且自动指标与人类判断对齐度低。

摘要:

自动生成的问题常存在表达不清或事实不准确等问题,需要对其质量进行可靠且全面的评估。人工评估在问题生成(QG)领域被广泛使用,并作为自动评估指标的黄金标准。然而,目前缺乏统一的人工评估标准,这阻碍了对QG模型和自动评估指标进行一致且可靠的评估。为此,我们提出QGEval,一个用于问题生成的多维度评估基准,从7个维度评估生成的问题和现有自动指标:流畅性、清晰度、简洁性、相关性、一致性、可回答性和答案一致性。我们通过考察这些维度的相关性和区别,证明了这些维度的适用性。通过使用QGEval对QG模型和自动指标进行一致评估,我们发现:1)大多数QG模型在可回答性和答案一致性方面表现不佳;2)在评估7个维度上生成的问题时,现有指标未能与人类判断良好对齐。我们期望这项工作能促进QG技术及其评估的发展。

英文摘要:

Automatically generated questions often suffer from problems such as unclear expression or factual inaccuracies, requiring a reliable and comprehensive evaluation of their quality. Human evaluation is widely used in the field of question generation (QG) and serves as the gold standard for automatic metrics. However, there is a lack of unified human evaluation criteria, which hampers consistent and reliable evaluations of both QG models and automatic metrics. To address this, we propose QGEval, a multi-dimensional Evaluation benchmark for Question Generation, which evaluates both generated questions and existing automatic metrics across 7 dimensions: fluency, clarity, conciseness, relevance, consistency, answerability, and answer consistency. We demonstrate the appropriateness of these dimensions by examining their correlations and distinctions. Through consistent evaluations of QG models and automatic metrics with QGEval, we find that 1) most QG models perform unsatisfactorily in terms of answerability and answer consistency, and 2) existing metrics fail to align well with human judgments when evaluating generated questions across the 7 dimensions. We expect this work to foster the development of both QG technologies and their evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑