评估评估指标：使用测量理论分析NLG评估指标的框架
Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory
AI总结：
提出基于测量理论的MetricEval框架，用于概念化和评估NLG评估指标的可靠性与有效性，并揭示了人工评估和基于LLM的指标中存在的问题。
摘要：
我们探讨了自然语言生成（NLG）模型评估中的一个根本挑战——评估指标的设计与评估。鉴于现有自动指标的局限性以及当前人工评估方式带来的噪声，我们提出了MetricEval，这是一个基于测量理论（教育测试设计的基础）的框架，用于概念化和评估NLG评估指标的可靠性与有效性。该框架将测量误差的来源形式化，并提供了基于实证数据评估评估指标的统计工具。借助我们的框架，可以量化指标的不确定性，从而更好地解释结果。为了示范框架的实际应用，我们分析了一组摘要评估指标，并发现了与人工评估中混淆的有效性结构以及基于LLM的指标的可靠性相关的问题。通过MetricEval，我们旨在促进有效且可靠的指标的设计、评估与解释，以推动稳健且有效的NLG模型的发展。
英文摘要：
We address a fundamental challenge in Natural Language Generation (NLG) model evaluation -- the design and evaluation of evaluation metrics. Recognizing the limitations of existing automatic metrics and noises from how current human evaluation was conducted, we propose MetricEval, a framework informed by measurement theory, the foundation of educational test design, for conceptualizing and evaluating the reliability and validity of NLG evaluation metrics. The framework formalizes the source of measurement error and offers statistical tools for evaluating evaluation metrics based on empirical data. With our framework, one can quantify the uncertainty of the metrics to better interpret the result. To exemplify the use of our framework in practice, we analyzed a set of evaluation metrics for summarization and identified issues related to conflated validity structure in human-eval and reliability in LLM-based metrics. Through MetricEval, we aim to promote the design, evaluation, and interpretation of valid and reliable metrics to advance robust and effective NLG models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
