大型语言模型尚不是抽象式摘要的人类水平评估器
Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization
AI总结：
本研究分析了ChatGPT和GPT-4等LLMs作为抽象式摘要自动评估器的稳定性和可靠性，发现它们虽优于传统自动指标，但因评分不一致、维度依赖及对高质量摘要评估相关性低等局限，尚不能替代人类评估。
摘要：
随着ChatGPT和GPT-4等大型语言模型（LLMs）在推理能力上近期取得的不可否认的进步，将LLMs用于各种任务的趋势日益增长。LLMs可被应用的一个领域是作为复杂生成任务的替代评估指标，这些任务通常需要昂贵的人工评判员来补充传统自动指标，以覆盖流畅性和一致性等各种评估维度。在本研究中，我们进行了广泛分析，以调查LLMs作为抽象式摘要自动评估器的稳定性和可靠性。我们发现，尽管ChatGPT和GPT-4的表现优于常用的自动指标，但由于存在显著局限性，它们尚未准备好替代人类。也就是说，LLM评估器对每个候选系统的评分不一致，且依赖于维度。它们也难以比较性能接近的候选者，并且对于更高质量的摘要，由于与人类的相关性降低而变得更加不可靠。换言之，随着更好的抽象式摘要系统被快速推出，LLMs可能会导致误导性和不可靠的评估。
英文摘要：
With the recent undeniable advancement in reasoning abilities in large language models (LLMs) like ChatGPT and GPT-4, there is a growing trend for using LLMs on various tasks. One area where LLMs can be employed is as an alternative evaluation metric for complex generative tasks, which generally demands expensive human judges to complement the traditional automatic metrics for various evaluation dimensions such as fluency and consistency. In this work, we conduct extensive analysis to investigate the stability and reliability of LLMs as automatic evaluators for abstractive summarization. We found that while ChatGPT and GPT-4 outperform the commonly used automatic metrics, they are not ready as human replacements due to significant limitations. That is, LLM evaluators rate each candidate system inconsistently and are dimension-dependent. They also struggle to compare candidates with close performance and become more unreliable with higher-quality summaries by obtaining a lower correlation with humans. In other words, with better abstractive summarization systems being introduced at a fast pace, LLMs may result in misleading and unreliable evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
