Large Language Models (LLMs) are increasingly serving as evaluators in Natural Language Generation (NLG) tasks; this is often referred to as ``LLM-as-a-judge'' paradigm. However, the capabilities of LLMs in evaluating NLG quality remain underexplored. Current studies depend on human assessments and simple metrics that fail to capture the discernment of LLMs across diverse NLG tasks. To address this gap, we propose the Discernment of Hierarchical Perturbation (DHP) benchmarking framework, which provides quantitative discernment scores for LLMs. This framework leverages hierarchically perturbed text data and statistical tests to systematically measure the NLG evaluation capabilities of LLMs. We re-established six evaluation datasets for this benchmark, covering four NLG tasks: Summarization, Story Completion, Question Answering, and Translation. Our comprehensive benchmarking of five major LLM families provides critical insight into their strengths and limitations as NLG evaluators. Our dataset is available at https://huggingface.co/datasets/YCWANGVINCE/DHP_Benchmark.
Unveiling the Achilles' Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models
National University of Singapore（新加坡国立大学）；ASUS Intelligent Cloud Services（华硕智能云服务）；Kriston AI Lab（科润人工智能实验室）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Universidad Politécnica de Madrid（马德里理工大学）
Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluators
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Data Science and Intelligent Media, Communication University of China（中国传媒大学数据科学与智能媒体学院）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Microsoft Cognitive Services Research（微软认知服务研究院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Carnegie Mellon University（卡内基梅隆大学）
A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators
National University of Singapore（新加坡国立大学）；University of Electronic Science and Technology of China（电子科技大学）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））
MEDLAYXPLAIN: Benchmarking the Expert-Lay Gap in Medical Vision-Language Models
Seoul National University（首尔大学）；Seoul National University Hospital（首尔大学医院）；Seoul National University College of Medicine（首尔大学医学院）；Sungkyunkwan University School of Medicine（成均馆大学医学院）
