As large-language models have been increasingly used as automatic raters for evaluating free-form content, including document summarization, dialog, and story generation, work has been dedicated to evaluating such models by measuring their correlations with human judgment. For \textit{sample-level} performance, methods which operate by using pairwise comparisons between machine-generated text perform well but often lack the ability to assign absolute scores to individual summaries, an ability crucial for use cases that require thresholding. In this work, we propose a direct-scoring method which uses synthetic summaries to act as pairwise machine rankings at test time. We show that our method performs comparably to state-of-the-art pairwise evaluators in terms of axis-averaged sample-level correlations on the SummEval (\textbf{+0.03}), TopicalChat (\textbf{-0.03}), and HANNA (\textbf{+0.05}) meta-evaluation benchmarks, and release the synthetic in-context summaries as data to facilitate future work.
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Microsoft Cognitive Services Research（微软认知服务研究院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Carnegie Mellon University（卡内基梅隆大学）
Unveiling the Achilles' Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models
National University of Singapore（新加坡国立大学）；ASUS Intelligent Cloud Services（华硕智能云服务）；Kriston AI Lab（科润人工智能实验室）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Universidad Politécnica de Madrid（马德里理工大学）
Multimodal Language Models as Text-to-Image Model Evaluators
FAIR at Meta - Montreal, New York（Meta 蒙特利尔 FAIR）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Mila；McGill University（麦吉尔大学）；Canada CIFAR AI chair（加拿大 CIFAR 人工智能主席）
Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluators
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Data Science and Intelligent Media, Communication University of China（中国传媒大学数据科学与智能媒体学院）
