Evaluating natural language generation (NLG) is a vital but challenging problem in natural language processing. Traditional evaluation metrics mainly capturing content (e.g. n-gram) overlap between system outputs and references are far from satisfactory, and large language models (LLMs) such as ChatGPT have demonstrated great potential in NLG evaluation in recent years. Various automatic evaluation methods based on LLMs have been proposed, including metrics derived from LLMs, prompting LLMs, fine-tuning LLMs, and human-LLM collaborative evaluation. In this survey, we first give a taxonomy of LLM-based NLG evaluation methods, and discuss their pros and cons, respectively. Lastly, we discuss several open problems in this area and point out future research directions.
Leveraging Large Language Models for NLG Evaluation: Advances and Challenges
Peking University(北京大学);Institute of Information Engineering, CAS(中国科学院信息工程研究所);AAII, FEIT, University of Technology Sydney(悉尼科技大学);UCLA(加利福尼亚大学洛杉矶分校);The Open University of China(国家开放大学);Beihang University(北京航空航天大学)
Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluators
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);School of Data Science and Intelligent Media, Communication University of China(中国传媒大学数据科学与智能媒体学院)
Unveiling the Achilles' Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models
National University of Singapore(新加坡国立大学);ASUS Intelligent Cloud Services(华硕智能云服务);Kriston AI Lab(科润人工智能实验室);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Universidad Politécnica de Madrid(马德里理工大学)
From Instruction to Output: The Role of Prompting in Modern NLG
Faculty of Information Technology, Monash University(莫纳什大学信息科技学院);School of Computer, Data and Mathematical Sciences, Western Sydney University(西澳大学计算机、数据与数学科学学院);School of Computing, Macquarie University(麦考瑞大学计算学院)
Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Carle’s Illinois College of Medicine, University of Illinois at Urbana-Champaign(卡尔伊利诺伊医学院,伊利诺伊大学厄巴纳-香槟分校)
LF-Steering: Latent Feature Activation Steering for Enhancing Semantic Consistency in Large Language Models
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);IT Innovation and Research Center, Huawei Technologies(华为技术有限公司 IT 创新研究中心);Informatics, University of Edinburgh(爱丁堡大学信息学院)
Evaluating, Understanding, and Improving Constrained Text Generation for Large Language Models
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Center for Data Science, Peking University(北京大学数据科学中心);The MOE Key Laboratory of Computational Linguistics, Peking University(北京大学教育部计算语言学重点实验室)
Do Large Language Models Adapt to Language Variation across Socioeconomic Status?
IT University of Copenhagen(丹麦技术大学);Pioneer Center for AI(先锋人工智能中心);University of Copenhagen(哥本哈根大学);Bocconi University(博科尼大学);CENTAI Institute(CENTAI研究所)
Transformer-Assisted LLM-Based Source Code Summarisation: to Enable More Secure Software Development
School of Computing and Communications, Lancaster University, UK(兰卡斯特大学计算与通讯学院);College of Engineering & Computer Science, VinUniversity(Vin大学工程与计算机科学学院)
DialogueLLM: Context and Emotion Knowledge-Tuned Large Language Models for Emotion Recognition in Conversations
The Hong Kong Polytechnic University(香港理工大学);Zhengzhou University of Light Industry(郑州轻工业大学);Hebei University of Technology(河北工业大学);Halmstad University(哈尔姆斯塔德大学);Copenhagen University(哥本哈根大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators
National University of Singapore(新加坡国立大学);University of Electronic Science and Technology of China(电子科技大学);The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳))