我们在NLG中测量什么？2020-2025年评估趋势的元分析
What Are We Measuring in NLG? A Meta-Analysis of Evaluation Trends 2020-2025
AI总结：
通过元分析14,171篇论文，揭示NLG评估中的三个系统性问题：度量惯性、度量-标准映射问题和验证差距，并提出最小评估清单。
摘要：
随着自然语言生成（NLG）主导现代NLP，可扩展评估仍然是一个关键瓶颈。因此，LLM作为评判者（LaaJ）的采用迅速加速，在2025年出现在比人工评估更多的论文中。这一关键转变促使对当前评估实践进行批判性分析。克服刚性关键词过滤和人工审查的限制，我们采用多LLM信息提取管道，从四大NLP会议（2020-2025）的14,171篇论文中收集结构化元数据。分析3,334篇过滤后的NLG论文，我们识别出三个系统性问题。（1）度量惯性：尽管转向开放式生成，传统词汇度量（BLEU、ROUGE）仍作为主要指标持续存在，通常与语义替代方案一起使用而非被取代。（2）度量-标准映射问题：我们的论文级共现数据显示，通用自动度量被用作质量的广泛代理，而未指定它们旨在评估文本生成的哪个维度。（3）验证差距：LaaJ在没有相应人工验证的情况下快速增长（少于8%的论文）。关键的是，虽然LaaJ与整体质量相关，但在流畅性等细粒度标准上一致性崩溃。为弥补这些差距，我们将发现提炼为一个最小评估清单，以指导度量选择、构念效度和LaaJ部署。
英文摘要：
As Natural Language Generation (NLG) dominates modern NLP, scalable evaluation remains a critical bottleneck. Consequently, LLM-as-a-judge (LaaJ) adoption has accelerated rapidly, appearing in more papers than human evaluation in 2025. This pivotal shift motivates a critical analysis of current evaluation practices. Overcoming the limits of rigid keyword filtering and manual review, we employ a multi-LLM information extraction pipeline to gather structured metadata from 14,171 papers across four major NLP conferences (2020-2025). Analyzing 3,334 filtered NLG papers, we identify three systemic challenges. (1) Metric inertia: despite the shift toward open-ended generation, legacy lexical metrics (BLEU, ROUGE) persist as primary indicators, typically used alongside rather than replaced by semantic alternatives. (2) Metric-criteria mapping problem: our paper-level co-occurrence data reveals that general-purpose automatic metrics are applied as broad proxies for quality, without specifying which dimension of text generation they are intended to evaluate. (3) Validation gap: LaaJ has grown rapidly without commensurate human validation (fewer than 8% of papers). Crucially, while LaaJ correlates with aggregate quality, alignment collapses on fine-grained criteria like fluency. To address these gaps, we distill our findings into a minimal Evaluation Checklist to guide metric selection, construct validity, and LaaJ deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
