CHI：一种统一实体、关系和数量维度的复合幻觉指数，用于摘要评估
CHI: A Composite Hallucination Index Unifying Entity, Relation, and Quantity Dimensions for Summarization Evaluation
AI总结：
针对现有摘要忠实性指标仅覆盖单一幻觉类型的问题，提出统一度量CHI，将错误分解为实体、关系、数量三个正交维度，经调和平均融合，在SummEval上相关性达0.66，优于现有基线。
AI中文摘要：
抽象摘要的忠实性评估仍然是一个开放的挑战，现有指标仅处理孤立的幻觉类型：事实实体错误、关系不一致或数值捏造，而未捕捉它们的共现或交互。我们引入了CHI（复合幻觉指数），这是第一个统一的幻觉度量，它将忠实性错误分解为三个正交维度：实体幻觉（EHI）、关系幻觉（RHI*）和数量幻觉（QHI）。每个维度采用共享的基于Venn图派生因素的softmax归一化架构，这些因素代表抽取性、正幻觉、过度聚焦、负幻觉和焦点丢失。新颖的QHI组件引入了具有精确、epsilon、派生和时间比较模式的容错感知数值匹配。我们通过调和平均融合这三个维度，产生一个惩罚任何维度弱点的单一复合分数。我们在涵盖四个领域（新闻、医学、法律、金融）的800篇源文章上验证了CHI，摘要来自五个生成系统。实证结果表明：（i）三个维度在统计上是正交的（平均rho = 0.148），确认它们捕获不同的错误类型；（ii）在SummEval上，CHI实现了与人类判断的最高系统级相关性（rho = 0.66，p = 0.006），优于ROUGE（rho = 0.53）、EHI（rho = 0.58）和所有单独组件；（iii）消融研究证实所有三个维度贡献了独特的方差，完整复合指标优于任何单独组件，同时提供单分数基线无法提供的可分解错误诊断。CHI为从业者提供了一种可分解、可解释且高效的忠实性度量，适用于摘要系统的离线评估和在线监控。
英文摘要：
Faithfulness evaluation of abstractive summaries remains an open challenge, with existing metrics addressing only isolated hallucination types: factual entity errors, relational inconsistencies, or numerical fabrications, without capturing their co-occurrence or interaction. We introduce CHI (Composite Hallucination Index), the first unified hallucination metric that decomposes faithfulness errors into three orthogonal dimensions: entity hallucination (EHI), relation hallucination (RHI*), and quantity hallucination (QHI). Each dimension employs a shared softmax-normalized architecture over Venn diagram-derived factors representing extractiveness, positive hallucination, over-focus, negative hallucination, and lost focus. The novel QHI component introduces tolerance-aware numerical matching with exact, epsilon, derived, and temporal comparison modes. We fuse the three dimensions via harmonic mean to produce a single composite score that penalizes weakness in any dimension. We validate CHI on 800 source articles spanning four domains (news, medical, legal, financial) with summaries from five generation systems. Empirical results demonstrate that: (i) the three dimensions are statistically orthogonal (mean rho = 0.148), confirming they capture distinct error types; (ii) CHI achieves the highest system-level correlation with human judgments (rho = 0.66, p = 0.006) on SummEval, outperforming ROUGE (rho = 0.53), EHI (rho = 0.58), and all individual components; and (iii) ablation studies confirm that all three dimensions contribute unique variance, with the full composite outperforming any individual component while providing decomposable error diagnostics unavailable from single-score baselines. CHI provides practitioners with a decomposable, interpretable, and efficient faithfulness metric suitable for both offline evaluation and online monitoring of summarization systems.
