QAScore --一种用于问题生成评价的无监督无参考度量
QAScore -- An Unsupervised Unreferenced Metric for the Question Generation Evaluation
摘要:
问题生成(QG)的目的是自动化的任务,撰写问题的一组选定的答案中发现的段落。近年来,神经生成模型的引入使得自动生成的问题在质量方面有了实质性的改进,特别是与采用手动制作的问答的传统方法相比。然而,QG评估中常用的指标因与人类判断的一致性低而受到批评。因此,我们提出了一个新的无参考的评估指标,有可能提供一个更好的机制,用于评估QG系统,称为QAScore。QAScore不是微调语言模型以最大化其与人类判断的相关性,而是通过根据语言模型可以正确地生成问题答案中的掩码的概率计算交叉熵来评估问题。此外,我们进行了一个新的众包人类评估实验的QG评估,调查QAScore和其他指标如何与人类的判断。实验表明,QAScore获得了更强的相关性与我们提出的人类评价方法的结果相比,现有的传统的基于词重叠的指标,如BLEU和ROUGE,以及现有的基于预训练模型的指标BERTScore。
英文摘要:
Question Generation (QG) aims to automate the task of composing questions for a passage with a set of chosen answers found within the passage. In recent years, the introduction of neural generation models has resulted in substantial improvements of automatically generated questions in terms of quality, especially compared to traditional approaches that employ manually crafted heuristics. However, the metrics commonly applied in QG evaluations have been criticized for their low agreement with human judgement. We therefore propose a new reference-free evaluation metric that has the potential to provide a better mechanism for evaluating QG systems, called QAScore. Instead of fine-tuning a language model to maximize its correlation with human judgements, QAScore evaluates a question by computing the cross entropy according to the probability that the language model can correctly generate the masked words in the answer to that question. Furthermore, we conduct a new crowd-sourcing human evaluation experiment for the QG evaluation to investigate how QAScore and other metrics can correlate with human judgements. Experiments show that QAScore obtains a stronger correlation with the results of our proposed human evaluation method compared to existing traditional word-overlap-based metrics such as BLEU and ROUGE, as well as the existing pretrained-model-based metric BERTScore.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。