arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.11503cs.CLcs.AI
收藏

为汇总评估找到平衡的自动化程度

Finding a Balanced Degree of Automation for Summary Evaluation

UNC Chapel Hill(北卡罗来纳大学教堂山分校)

Shiyue Zhang, Mohit Bansal

摘要:

摘要任务的人工评估是可靠的,但带来了可重复性和高成本的问题。自动化的指标是廉价和可重复的,但有时与人类判断的相关性很差。在这项工作中,我们提出了灵活的半自动摘要评价指标,金字塔人类评价方法。半自动Lite 2 Pyramid保留了可重复使用的人工标记的摘要内容单元(SCU)供参考,但用自然语言推理(NLI)模型取代了判断SCU是否存在于系统摘要中的手动工作。全自动Lite 3 Pyramid进一步通过语义角色标注(SRL)模型用自动提取的语义三元组单元(STU)替换SCU。最后,我们提出了中间指标Lite2.xPyramid,其中我们使用一个简单的回归器来预测STU模拟SCU的效果,并保留更难以模拟的SCU,这提供了自动化和手动评估之间的平滑过渡和平衡。与现有的15个指标相比,我们在3个现有的元评估数据集和我们新收集的PyrXSum(100/10 XSum示例/系统)上评估了人类指标的相关性。它表明Lite 2 Pyramid始终具有最佳的摘要级相关性; Lite 3 Pyramid比其他自动度量更好或可与之媲美; Lite2.xPyramid权衡了较小的相关性下降,以减少更大的手动工作量,这可以降低未来数据汇集的成本。我们的代码和数据可在以下网址公开获取:https://github.com/ZhangShiyue/Lite2-3Pyramid

英文摘要:

Human evaluation for summarization tasks is reliable but brings in issues of reproducibility and high costs. Automatic metrics are cheap and reproducible but sometimes poorly correlated with human judgment. In this work, we propose flexible semiautomatic to automatic summary evaluation metrics, following the Pyramid human evaluation method. Semi-automatic Lite2Pyramid retains the reusable human-labeled Summary Content Units (SCUs) for reference(s) but replaces the manual work of judging SCUs' presence in system summaries with a natural language inference (NLI) model. Fully automatic Lite3Pyramid further substitutes SCUs with automatically extracted Semantic Triplet Units (STUs) via a semantic role labeling (SRL) model. Finally, we propose in-between metrics, Lite2.xPyramid, where we use a simple regressor to predict how well the STUs can simulate SCUs and retain SCUs that are more difficult to simulate, which provides a smooth transition and balance between automation and manual evaluation. Comparing to 15 existing metrics, we evaluate human-metric correlations on 3 existing meta-evaluation datasets and our newly-collected PyrXSum (with 100/10 XSum examples/systems). It shows that Lite2Pyramid consistently has the best summary-level correlations; Lite3Pyramid works better than or comparable to other automatic metrics; Lite2.xPyramid trades off small correlation drops for larger manual effort reduction, which can reduce costs for future data collection. Our code and data are publicly available at: https://github.com/ZhangShiyue/Lite2-3Pyramid

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑