KPEval：迈向基于细粒度语义的关键短语评估
KPEval: Towards Fine-Grained Semantic-Based Keyphrase Evaluation
AI总结：
提出KPEval框架，通过参考一致性、忠实度、多样性和效用四个基于语义的指标评估关键短语系统，发现现有模型比较存在盲点且大语言模型被低估。
摘要：
尽管关键短语提取和关键短语生成方法取得了显著进展，但主要的评估方法仍主要依赖于与人工参考的精确匹配。这种方案无法识别生成与参考语义等价的关键短语或具有实际效用的多样化关键短语的系统。为了更好地评估关键短语系统的能力，我们提出了KPEval，这是一个全面的评估框架，包含四个关键方面：参考一致性、忠实度、多样性和效用。对于每个方面，我们设计了基于语义的指标以反映评估目标。元评估研究表明，与一系列先前提出的指标相比，我们的评估策略与人类偏好具有更好的相关性。使用KPEval，我们重新评估了23个关键短语系统，发现：（1）既定的模型比较结果存在盲点，特别是在考虑无参考评估时；（2）大型语言模型被先前的评估工作低估了；（3）没有单一的最佳模型能够在所有方面都表现优异。
英文摘要：
Despite the significant advancements in keyphrase extraction and keyphrase generation methods, the predominant approach for evaluation mainly relies on exact matching with human references. This scheme fails to recognize systems that generate keyphrases semantically equivalent to the references or diverse keyphrases that carry practical utility. To better assess the capability of keyphrase systems, we propose KPEval, a comprehensive evaluation framework consisting of four critical aspects: reference agreement, faithfulness, diversity, and utility. For each aspect, we design semantic-based metrics to reflect the evaluation objectives. Meta-evaluation studies demonstrate that our evaluation strategy correlates better with human preferences compared to a range of previously proposed metrics. Using KPEval, we re-evaluate 23 keyphrase systems and discover that (1) established model comparison results have blind-spots especially when considering reference-free evaluation; (2) large language models are underestimated by prior evaluation works; and (3) there is no single best model that can excel in all the aspects.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
