评估抽象式摘要中抽象性与事实性之间的权衡
Evaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarization
AI总结：
针对抽象式摘要中抽象性提升常损害事实性的问题，本文通过人类评估分析二者权衡，发布两个判断数据集并提出抽象性调整的事实性指标与基线。
摘要：
用于抽象式摘要的神经模型倾向于生成流畅且结构良好的输出，但相对于输入文档缺乏语义忠实性，即事实性。本文通过广泛的人类事实性评估，分析了多个数据集和模型上生成摘要的抽象性与事实性之间的权衡。在分析中，我们可视化了在使用解码约束逐步提高抽象性时事实性的变化率，并观察到，虽然抽象性提高通常会导致事实性下降，但事实性衰减的速度取决于系统所训练的数据等因素。我们引入了两个人工事实性判断数据集：一个包含1.02万条抽象程度系统变化的生成摘要；另一个包含来自五个不同摘要模型的4200条摘要。我们提出了根据抽象程度进行调整的新事实性指标，并用它们比较以往摘要工作的经抽象性调整的事实性，为未来工作提供基线。
英文摘要：
Neural models for abstractive summarization tend to generate output that is fluent and well-formed but lacks semantic faithfulness, or factuality, with respect to the input documents. In this paper, we analyze the tradeoff between abstractiveness and factuality of generated summaries across multiple datasets and models, using extensive human evaluations of factuality. In our analysis, we visualize the rates of change in factuality as we gradually increase abstractiveness using a decoding constraint, and we observe that, while increased abstractiveness generally leads to a drop in factuality, the rate of factuality decay depends on factors such as the data that the system was trained on. We introduce two datasets with human factuality judgements; one containing 10.2k generated summaries with systematically varied degrees of abstractiveness; the other containing 4.2k summaries from five different summarization models. We propose new factuality metrics that adjust for the degree of abstractiveness, and we use them to compare the abstractiveness-adjusted factuality of previous summarization works, providing baselines for future work.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
