Current developments in large language models (LLMs) have enabled impressive zero-shot capabilities across various natural language tasks. An interesting application of these systems is in the automated assessment of natural language generation (NLG), a highly challenging area with great practical benefit. In this paper, we explore two options for exploiting the emergent abilities of LLMs for zero-shot NLG assessment: absolute score prediction, and comparative assessment which uses relative comparisons between pairs of candidates. Though comparative assessment has not been extensively studied in NLG assessment, we note that humans often find it more intuitive to compare two options rather than scoring each one independently. This work examines comparative assessment from multiple perspectives: performance compared to absolute grading; positional biases in the prompt; and efficient ranking in terms of the number of comparisons. We illustrate that LLM comparative assessment is a simple, general and effective approach for NLG assessment. For moderate-sized open-source LLMs, such as FlanT5 and Llama2-chat, comparative assessment is superior to prompt scoring, and in many cases can achieve performance competitive with state-of-the-art methods. Additionally, we demonstrate that LLMs often exhibit strong positional biases when making pairwise comparisons, and we propose debiasing methods that can further improve performance.
"You are an expert annotator": Automatic Best-Worst-Scaling Annotations for Emotion Intensity Modeling
Merck Data & AI Organization, Merck Group(默克集团默克数据与人工智能部门);Institut für Maschinelle Sprachverarbeitung, University of Stuttgart(斯图加特大学机器语言处理研究所);Fundamentals of Natural Language Processing, University of Bamberg(班贝格大学自然语言处理基础研究所);Merck IT Centre, Merck Group(默克集团默克IT中心)
Blinded Radiologist and LLM-Based Evaluation of LLM-Generated Japanese Translations of Chest CT Reports: Comparative Study
Division of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo(东京大学医学研究生院放射学与生物医学工程系);Department of Radiology, The University of Tokyo Hospital(东京大学医院放射科);Department of Radiology, Kanazawa University Graduate School of Medical Sciences(金泽大学医学研究生院放射科);Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital(东京大学医院计算诊断放射学与预防医学系);Department of Radiology, School of Medicine, Jichi Medical University(自治医科大学医学院放射科)
A Comparative Empirical Study of Catastrophic Forgetting Mitigation in Sequential Task Adaptation for Continual Natural Language Processing Systems
Zaven P. and Sonia Akian College of Science and Engineering(泽文·皮和索尼亚·阿基安科学与工程学院);American University of Armenia(亚美尼亚美国大学);Yerevan(亚美尼亚共和国首都耶莱范);Armenia(亚美尼亚)