用于抽象摘要中关系级幻觉评估的基于基础与分解的框架
A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization
浏览论文内容
中文总结 AI 辅助
本研究提出一种用于抽象摘要关系级幻觉评估的基于基础与分解的框架，引入关系幻觉指数（RHI）及其归一化公式，经多模型评估验证其可生成稳定且具区分性的幻觉测量结果，推进了自动化关系级保真度评估。
中文摘要 AI 辅助
抽象文本摘要系统常生成流畅但不忠实的摘要，方法是编造或歪曲实体与事件间的关系。这种关系级幻觉会破坏生成摘要的可靠性，尤其在高风险领域。本研究提出一种用于评估抽象摘要中关系幻觉的精细且基于基础的框架。我们引入经验性关系幻觉指数（RHI），该指数结合了基于词形归并的归一化、基于命名实体的基础主语消解、被动施事恢复、感知否定的动词建模、引述动词过滤、名词性关系回退、分句传播及系统去重的依赖感知关系抽取算法。这些改进提升了抽取的关系三元组的结构保真度，减少了评估过程中的虚假匹配。此外，我们提出RHI的归一化公式，以确保数据集与模型间的尺度不变比较。修订后的指标将幻觉分解为可解释的组件，将关系幻觉指标聚合为归一化的关系保真度分数。对多个最先进摘要模型的广泛评估表明，基于基础的抽取过程能产生更稳定且具区分性的幻觉测量结果。所提出的框架推进了自动化关系级保真度评估，并支持感知连贯性、对幻觉敏感的模型分析。
英文摘要
Abstractive text summarization systems frequently generate fluent yet unfaithful summaries by fabricating or distorting relationships between entities and events. Such relation-level hallucinations undermine the reliability of generated summaries, particularly in high-stakes domains. In this work, we present a refined and grounded framework for evaluating relation hallucination in abstractive summarization. We present the empirical Relation Hallucination Index (RHI) by introducing a dependency-aware relation extraction algorithm that incorporates lemmatization-based normalization, named entity grounded subject resolution, passive agent recovery, negation-aware verb modeling, reporting verb filtering, nominal relation fallback, clausal propagation, and systematic deduplication. These enhancements improve the structural fidelity of extracted relation triples and reduce spurious matches during evaluation. In addition, we introduce a normalized formulation of RHI to ensure scale-invariant comparison between datasets and models. The revised metric decomposes hallucination into interpretable components, aggregates relation hallucination metric into a normalized relation faithfulness score. Extensive evaluation across multiple state-of-the-art summarization models demonstrates that the grounded extraction process yields more stable and discriminative hallucination measurements. The proposed framework advances automated relation-level faithfulness evaluation and supports coherence-aware, hallucination-sensitive model analysis.
发表机构
- IIIT Bhubaneswar（布巴内斯瓦尔印度信息技术学院）
- Informatica（Informatica公司）
- TEKsystems Global Services（TEKsystems全球服务公司）
机构由 AI 辅助整理，请以论文原文为准。
