出色的指标及其发现方法：多语言风格个性化生成指标的元评估
Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation
AI总结：
本研究针对多语言风格个性化生成指标，提出基于回归的元评估框架，构建大规模SPTG基准，发现经人类评分校准的简单指标可媲美LLM评判器，还明确了指标性能相关的标注特征与集成优势条件。
摘要：
文本生成指标与大语言模型（LLM）评判器的元评估主要通过指标与人类评判的相关性开展。但已有研究表明，相关性并非衡量指标评估能力的合适指标，且在不同评估设置（如不同数据集或语言）中会表现不稳定。因此，我们提出一种基于回归的指标元评估方法，该方法利用校准后的指标输出来预测人类评分。我们以简单的均值基线和人类一致性为基准，对该测量方法进行校准，从而在不同评估设置下提供可比较的指标能力衡量标准。为测试我们的元评估框架，我们构建了一个大规模多语言风格个性化文本生成（SPTG）基准，收集了10种拓扑多样性语言及4个评估维度的多轮标注，各维度具有不同程度的主观性。我们在这40种评估设置下，对来自3种风格评估范式（基于字符串、基于嵌入、LLM评判器）的32个指标及其集成模型进行了元评估。分析显示，经人类评分校准后，即使是简单的基于字符串和嵌入的测量也能与强大的LLM评判器相媲美。我们还明确了决定指标性能的人类标注特征，以及指标集成模型优于其单个组成部分的条件。
英文摘要：
Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that correlation is not an appropriate estimate of a metric's evaluation capability and can be unstable across evaluation settings, such as different datasets or languages. Therefore, we propose a regression-based meta-evaluation of metrics that learns on calibrated metric outputs to predict human ratings. We ground this measurement relative to a trivial constant-mean baseline and human agreements to provide a comparable measure of metric capability across evaluation settings. To test our meta-evaluation framework, we developed a large-scale multi-lingual style-personalized text generation (SPTG) benchmark, collecting multiple annotations across 10 topologically diverse languages and four evaluation dimensions, each with varying degrees of subjectivity. We meta-evaluated 32 metrics from three style evaluation paradigms (string-based, embedding-based, and LLM-judges) and their ensembles over these 40 evaluation settings. Our analysis revealed that even simple string- and embedding-based measurements can match strong LLM-judges when calibrated to human ratings. We also identified characteristics of human annotations that determine metric performance, as well as conditions under which a metric ensemble outperforms its individual constituents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
