PerSEval：评估文本摘要器中的个性化程度
PerSEval: Assessing Personalization in Text Summarizers
AI总结：
本文指出 EGISES 仅衡量响应性而非充分的个性化程度，提出 PerSEval，并在 PENS 上对十个 SOTA 模型验证其与人类判断的相关性、排名稳定性及独立评估价值。
摘要：
个性化摘要模型会迎合个体对显著性的主观理解，这种理解由其阅读历史和当前关注主题所体现。现有的个性化文本摘要器主要依据 BLEU、ROUGE 和 METEOR 等准确性指标进行评估。然而，近期一项研究认为，准确性指标不足以评估这些模型的个性化程度，并提出了 EGISES，即首个用于评估个性化文本摘要的指标。该研究提出，准确性是一个独立方面，应当单独评估。本文对准确性排行榜的必要性提出质疑，认为依赖基于准确性的聚合结果可能导致误导性结论。为支持这一观点，我们更深入地研究 EGISES，从理论和实证两方面证明，它衡量的是响应程度，而响应程度只是个性化程度的必要条件而非充分条件。随后，我们提出 PerSEval，这是一种满足所需充分性条件的新型度量。基于在 PENS 数据集上对十个 SOTA 摘要模型进行的基准测试，我们通过实证确定：（i）PerSEval 在与人类判断的相关性方面是可靠的（Pearson's r = 0.73；Spearman's ρ = 0.62；Kendall's τ = 0.42），（ii）PerSEval 具有较高的排名稳定性，（iii）作为排名度量的 PerSEval 并不被基于 EGISES 的排名所蕴含，以及（iv）PerSEval 可以作为独立的排名度量，而无需任何聚合排名。
英文摘要：
Personalized summarization models cater to individuals' subjective understanding of saliency, as represented by their reading history and current topics of attention. Existing personalized text summarizers are primarily evaluated based on accuracy measures such as BLEU, ROUGE, and METEOR. However, a recent study argued that accuracy measures are inadequate for evaluating the degree of personalization of these models and proposed EGISES, the first metric to evaluate personalized text summaries. It was suggested that accuracy is a separate aspect and should be evaluated standalone. In this paper, we challenge the necessity of an accuracy leaderboard, suggesting that relying on accuracy-based aggregated results might lead to misleading conclusions. To support this, we delve deeper into EGISES, demonstrating both theoretically and empirically that it measures the degree of responsiveness, a necessary but not sufficient condition for degree-of-personalization. We subsequently propose PerSEval, a novel measure that satisfies the required sufficiency condition. Based on the benchmarking of ten SOTA summarization models on the PENS dataset, we empirically establish that -- (i) PerSEval is reliable w.r.t human-judgment correlation (Pearson's r = 0.73; Spearman's $ρ$ = 0.62; Kendall's $τ$ = 0.42), (ii) PerSEval has high rank-stability, (iii) PerSEval as a rank-measure is not entailed by EGISES-based ranking, and (iv) PerSEval can be a standalone rank-measure without the need of any aggregated ranking.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
