陀螺还在转吗？评估叙事理解中的主观性
Is the Top Still Spinning? Evaluating Subjectivity in Narrative Understanding
AI总结：
针对歧义主张忠实性评估的主观性问题，提出利用LLM编辑摘要的Ambiguity Rewrite Metric (ARM)指标，在叙事摘要领域使标注者一致性绝对提升21%，有效降低主观性。
摘要：
确定某个主张对源文档的忠实性是许多领域的重要问题。该任务通常被视为判断主张相对于源文档是受支持还是不受支持的二元判断。然而在许多情况下，主张是否受支持可能存在歧义。例如，它可能依赖于从给定证据进行推断，而不同的人可能基于对这些推断的认同，合理地将该主张解释为受支持或不受支持。对此类主张强制施加二元标签会降低评估的可靠性。在本研究中，我们重新构建任务以管理歧义主张事实性判断中的主观性。我们引入LLM生成的摘要编辑作为一种提供细致评估主张的方法：摘要需要被编辑多少才能消除歧义？主张是否被重写及其变化程度可用作自动评估指标——Ambiguity Rewrite Metric (ARM)，其反馈信号比二元忠实性判断丰富得多。我们聚焦于叙事摘要领域，因为该领域尤其充斥歧义和主观解释。我们表明，ARM在主张忠实性的标注者一致性上产生21%的绝对提升，表明主观性得到降低。
英文摘要：
Determining faithfulness of a claim to a source document is an important problem across many domains. This task is generally treated as a binary judgment of whether the claim is supported or unsupported in relation to the source. In many cases, though, whether a claim is supported can be ambiguous. For instance, it may depend on making inferences from given evidence, and different people can reasonably interpret the claim as either supported or unsupported based on their agreement with those inferences. Forcing binary labels upon such claims lowers the reliability of evaluation. In this work, we reframe the task to manage the subjectivity involved with factuality judgments of ambiguous claims. We introduce LLM-generated edits of summaries as a method of providing a nuanced evaluation of claims: how much does a summary need to be edited to be unambiguous? Whether a claim gets rewritten and how much it changes can be used as an automatic evaluation metric, the Ambiguity Rewrite Metric (ARM), with a much richer feedback signal than a binary judgment of faithfulness. We focus on the area of narrative summarization as it is particularly rife with ambiguity and subjective interpretation. We show that ARM produces a 21% absolute improvement in annotator agreement on claim faithfulness, indicating that subjectivity is reduced.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
