Validating evaluation metrics for NLG typically relies on expensive and time-consuming human annotations, which predominantly exist only for English datasets. We propose LLM as a Meta-Judge, a scalable framework that utilizes LLMs to generate synthetic evaluation datasets via controlled semantic degradation of real data, replacing human judgment. We validate our approach using meta-correlation, measuring the alignment between metric rankings derived from synthetic data and those from standard human benchmarks. Experiments across Machine Translation, Question Answering, and Summarization demonstrate that synthetic validation serves as a reliable proxy for human judgment, achieving meta-correlations exceeding 0.9 in multilingual QA and proves to be a viable alternative where human judgments are unavailable or too expensive to obtain. Our code and data are publicly available at https://github.com/eiglerl/meta-judge.
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
University of Amsterdam（阿姆斯特丹大学）；University of Trento（特伦托大学）；University of Copenhagen（哥本哈根大学）；Utrecht University（乌得勒支大学）；Max Planck Institute for Psycholinguistics（马克斯·普朗克心理语言学研究所）；ETH Zürich（苏黎世联邦理工学院）；Saarland University（萨尔大学）；Universidade de Lisboa & Unbabel（里斯本大学及Unbabel）；LMU Munich & MCML（慕尼黑大学及MCML）；University of Potsdam（波茨坦大学）；Heriot-Watt University（赫瑞线学院）
Generative AI Research Lab (GAIR)（生成式人工智能研究实验室（GAIR））；Hong Kong Polytechnic University（香港理工大学）；New York University（纽约大学）；Chinese Academy of Sciences（中国科学院）；Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation
The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University（清华大学对话人工智能（CoAI）小组）；Zhipu AI（智谱AI）；University of Electronic Science and Technology of China（电子科技大学）
Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English
HiTZ Center, University of the Basque Country (UPV/EHU)（巴斯克大学（UPV/EHU）HiTZ中心）；University of Alicante (UA)（阿利坎特大学（UA））；GOI Institute, Basque Summer University (UEU)（巴斯克夏季大学（UEU）GOI研究所）
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge
Arizona State University（亚利桑那州立大学）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）；University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）；Northwestern University（西北大学）；University of California, Berkeley（加州大学伯克利分校）；Emory University（埃默里大学）
MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models
Yonsei University（延世大学）；KAIST（韩国科学技术院）；OneLineAI；Barcelona Supercomputing Center（巴塞罗那超级计算中心）；Artful Media；Bangladesh University of Engineering and Technology（孟加拉国工程技术大学）；Carnegie Mellon University（卡内基梅隆大学）
MetaHOPE: A Metaphor-Oriented Evaluation Framework for Analysing MT and LLM Translation Errors
Centre for Linguistics, Humanities, Leiden University（莱顿大学人文学院语言学中心）；LIACS, Leiden University（莱顿大学LIACS）；BDS, Leiden University Medical Centre（莱顿大学医学中心BDS）
LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
University of Illinois Chicago（伊利诺伊大学芝加哥分校）；Salesforce Research（Salesforce研究院）；Hong Kong Polytechnic University（香港理工大学）；Penn State University（宾夕法尼亚州立大学）；Google DeepMind（谷歌DeepMind）；University of Southern California（南加州大学）；University of California, Davis（加州大学戴维斯分校）；University of Basel（巴塞尔大学）；Xiaomi AI Lab（小米AI实验室）；Scale AI；HKUST（香港科技大学）；Georgia Institute of Technology（佐治亚理工学院）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；Singapore University of Technology and Design（新加坡科技设计大学）；Texas A&M University（德克萨斯农工大学）；Massey University（梅西大学）；University of Liverpool（利物浦大学）
Meta-Judging with Large Language Models: Concepts, Methods, and Challenges
University of Coimbra, CISUC/LASI – Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering（科英布拉大学，CISUC/LASI——科英布拉大学信息与系统研究中心，信息工程系）；Polytechnic University of Coimbra, Rua da Misericórdia, Lagar dos Cortiços, S. Martinho do Bispo, 3045-093 Coimbra, Portugal（科英布拉理工大学）
Towards Multimodal Metaphor Understanding: A Chinese Dataset and Model for Metaphor Mapping Identification
Dalian University of Technology（大连理工大学）；School of Foreign Languages（外国语学院）；School of Software（软件学院）；University of Macau（澳门大学）；Faculty of Business Administration（工商管理学院）；Kansai University（关西大学）；Faculty of Business and Commerce（商学部）；University of New South Wales（新南威尔士大学）；School of Minerals and Energy Resources Engineering（矿产与能源资源工程学院）；RMIT University（皇家墨尔本理工大学）；Centre for Educational Innovation and Quality（教育创新与质量中心）；School of Computing Technologies（计算技术学院）
Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation
Department of Computer Science and Engineering, Islamic University of Technology, Gazipur, Bangladesh（计算机科学与工程系，伊斯兰技术大学，加兹ipur，孟加拉国）；Department of Computer Science（计算机科学系）；Engineering, Islamic University of Technology, Gazipur, Bangladesh（工程，伊斯兰技术大学，加兹ipur，孟加拉国）
Can Agents Judge Systematic Reviews Like Humans? Evaluating SLRs with LLM-based Multi-Agent System
Department of Computer Science（计算机科学系）；Information Technology University（信息技术大学）；AI Center for Precision Health（精准健康人工智能中心）；Weill Cornell Medicine–Qatar（韦尔·科恩医学中心–卡塔尔）；Computer Science and Engineering Department（计算机科学与工程系）
