UniSumEval：面向大语言模型的统一、细粒度、多维摘要评估
UniSumEval: Towards Unified, Fine-Grained, Multi-Dimensional Summarization Evaluation for LLMs
AI总结：
本文提出UniSumEval基准，通过AI辅助构建细粒度、多维标注并扩展输入场景，系统评估九种最新LLM的摘要能力及SOTA自动评估器。
摘要：
现有的摘要质量评估基准往往缺乏多样化的输入场景，仅关注狭义定义的维度（如忠实性），并且受制于主观且粗粒度的标注方案。为解决这些不足，我们构建了UniSumEval基准，该基准扩展了输入上下文的范围（如领域、长度），并提供细粒度、多维度的标注。我们在数据创建过程中使用AI辅助，识别可能产生幻觉的输入文本，同时帮助人类标注者降低细粒度标注任务的难度。借助UniSumEval，我们对九种最新语言模型作为摘要器进行了基准测试，深入分析了它们在不同输入上下文和评估维度上的表现。此外，我们对最先进的自动摘要评估器进行了全面比较。我们的基准数据将在https://github.com/DISL-Lab/UniSumEval-v1.0上公开。
英文摘要：
Existing benchmarks for summarization quality evaluation often lack diverse input scenarios, focus on narrowly defined dimensions (e.g., faithfulness), and struggle with subjective and coarse-grained annotation schemes. To address these shortcomings, we create UniSumEval benchmark, which extends the range of input context (e.g., domain, length) and provides fine-grained, multi-dimensional annotations. We use AI assistance in data creation, identifying potentially hallucinogenic input texts, and also helping human annotators reduce the difficulty of fine-grained annotation tasks. With UniSumEval, we benchmark nine latest language models as summarizers, offering insights into their performance across varying input contexts and evaluation dimensions. Furthermore, we conduct a thorough comparison of SOTA automated summary evaluators. Our benchmark data will be available at https://github.com/DISL-Lab/UniSumEval-v1.0.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
