Some prior work has shown that LLMs perform well in NLG evaluation for different tasks. However, we discover that LLMs seem to confuse different evaluation criteria, which reduces their reliability. For further verification, we first consider avoiding issues of inconsistent conceptualization and vague expression in existing NLG quality criteria themselves. So we summarize a clear hierarchical classification system for 11 common aspects with corresponding different criteria from previous studies involved. Inspired by behavioral testing, we elaborately design 18 types of aspect-targeted perturbation attacks for fine-grained analysis of the evaluation behaviors of different LLMs. We also conduct human annotations beyond the guidance of the classification system to validate the impact of the perturbations. Our experimental results reveal confusion issues inherent in LLMs, as well as other noteworthy phenomena, and necessitate further research and improvements for LLM-based evaluation.
Unveiling the Achilles' Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models
National University of Singapore（新加坡国立大学）；ASUS Intelligent Cloud Services（华硕智能云服务）；Kriston AI Lab（科润人工智能实验室）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Universidad Politécnica de Madrid（马德里理工大学）
Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluators
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Data Science and Intelligent Media, Communication University of China（中国传媒大学数据科学与智能媒体学院）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Microsoft Cognitive Services Research（微软认知服务研究院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Carnegie Mellon University（卡内基梅隆大学）
Multi-dimensional evaluation is the dominant paradigm for human evaluation in Natural Language Generation (NLG), i.e., evaluating the generated text from multiple explainable dimensions, such as coherence and fluency. However, automatic evaluation in NLG is still dominated by similarity-based metrics, and we lack a reliable framework for a more comprehensive evaluation of advanced models. In this paper, we propose a unified multi-dimensional evaluator UniEval for NLG. We re-frame NLG evaluation as a Boolean Question Answering (QA) task, and by guiding the model with different questions, we can…
A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators
National University of Singapore（新加坡国立大学）；University of Electronic Science and Technology of China（电子科技大学）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））
Institute of Natural Language Processing, University of Stuttgart（斯图加特大学自然语言处理研究所）；Konkuk University（建国大学）
Natural Language Generation (NLG) refers to the operation of expressing the calculation results of a system in human language. Since the quality of generated sentences from an NLG model cannot be fully represented using only quantitative evaluation, they are evaluated using qualitative evaluation by humans in which the meaning or grammar of a sentence is scored according to a subjective criterion. Nevertheless, the existing evaluation methods have a problem as a large score deviation occurs depending on the criteria of evaluators. In this paper, we propose Grammar Accuracy Evaluation (GAE) tha…
Quality of Answers of Generative Large Language Models vs Peer Patients for Interpreting Lab Test Results for Lay Patients: Evaluation Study
School of Information, Florida State University（佛罗里达州立大学信息学院）；National Center for Biotechnology Information (NCBI), National Library of Medicine (NLM), National Institutes of Health（美国国立卫生研究院国家医学图书馆国家生物技术信息中心）；Morsani College of Medicine, University of South Florida（南佛罗里达大学莫尔萨尼医学院）
