We describe GEMBA, a GPT-based metric for assessment of translation quality, which works both with a reference translation and without. In our evaluation, we focus on zero-shot prompting, comparing four prompt variants in two modes, based on the availability of the reference. We investigate nine versions of GPT models, including ChatGPT and GPT-4. We show that our method for translation quality assessment only works with GPT~3.5 and larger models. Comparing to results from WMT22's Metrics shared task, our method achieves state-of-the-art accuracy in both modes when compared to MQM-based human labels. Our results are valid on the system level for all three WMT22 Metrics shared task language pairs, namely English into German, English into Russian, and Chinese into English. This provides a first glimpse into the usefulness of pre-trained, generative large language models for quality assessment of translations. We publicly release all our code and prompt templates used for the experiments described in this work, as well as all corresponding scoring results, to allow for external validation and reproducibility.
MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators
College of Computing and Data Science at Nanyang Technological University（南洋理工大学计算与数据科学学院）；Southeast University（东南大学）；The University of Sydney（悉尼大学）；Southeast University Shenzhen Research Institute（东南大学深圳研究院）
Evaluate What You Can't Evaluate: Unassessable Quality for Generated Response
Northeastern University（东北大学）；Center for Information and Language Processing, LMU Munich（慕尼黑大学信息与语言处理中心）；Munich Center for Machine Learning (MCML), LMU Munich（慕尼黑大学慕尼黑机器学习中心）
Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple LLM Judges
ZJU–UIUC Institute（浙大UIUC研究院）；ZJU–Hangzhou Global Scientific and Technological Innovation Center（浙大杭州全球科技创新中心）；College of Computer Science and Technology（计算机科学与技术学院）；Alibaba Group（阿里巴巴集团）
Encoder-decoder has been widely used in neural machine translation (NMT). A few methods have been proposed to improve it with multiple passes of decoding. However, their full potential is limited by a lack of appropriate termination policies. To address this issue, we present a novel architecture, Rewriter-Evaluator. It consists of a rewriter and an evaluator. Translating a source sentence involves multiple passes. At every pass, the rewriter produces a new translation to improve the past translation and the evaluator estimates the translation quality to decide whether to terminate the rewriti…
GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks
University of California, Santa Barbara（加州大学圣塔芭芭拉分校）；University of California, San Diego（加州大学圣迭戈分校）；University of Southern California（南加州大学）；Bytedance, US（字节跳动（美国））
Fine-grained Gender Control in Machine Translation with Large Language Models
LG AI Research（LG AI研究院）；Seoul National University（首尔大学）；IPAI, Seoul National University（首尔大学IPAI）；Institute of Engineering Research, Seoul National University（首尔大学工程研究所）
Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
Department of Machine Learning, Carnegie Mellon University, Pittsburgh, PA, USA（卡内基梅隆大学机器学习系）；Department of Computer Science and Engineering, University of California San Diego, La Jolla, CA, USA（加州大学圣地亚哥分校计算机科学与工程系）；Department of Computer Science, University of Virginia, Charlottesville, VA, USA（弗吉尼亚大学计算机科学系）；Martian Research, San Francisco, California, USA（火星研究公司）；Apart Research, San Francisco, California, USA（Apart研究公司）
From Utterance to Vividity: Training Expressive Subtitle Translation LLM via Adaptive Local Preference Optimization
MAIS, Institute of Automation, Chinese Academy of Sciences（自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Hujing Digital Media & Entertainment Group（华景数字媒体与娱乐集团）；Geely AI lab（吉利AI实验室）
Quality estimation (QE) is the task of automatically evaluating the quality of translations without human-translated references. Calculating BLEU between the input sentence and round-trip translation (RTT) was once considered as a metric for QE, however, it was found to be a poor predictor of translation quality. Recently, various pre-trained language models have made breakthroughs in NLP tasks by providing semantically meaningful word and sentence embeddings. In this paper, we employ semantic embeddings to RTT-based QE. Our method achieves the highest correlations with human judgments, compar…
Distinct Profiles of Run-to-Run Score Reliability and Expert-Panel Alignment Across Four LLM Evaluators of Simulated Japanese-Language AI-to-AI Counseling
Japan National Institute of Occupational Safety and Health（日本国立职业安全卫生研究所）；Kaze To Taiyo（凯泽・太阳）；Saga Occupational Health Association（Saga职业健康协会）；Department of Pharmacy, Zikei Hospital/Zikei Institute of Psychiatry（药剂科，Zikei医院/Zikei精神医学研究所）；Department of Medical Welfare, Suzuka University of Medical Science（医疗福祉科， Suzuka医科大学）；Graduate School of Human Sciences, Ritsumeikan University（人类科学研究生院，立命馆大学）；Faculty of Nursing, National Defense Medical College（护理学部，国家防卫医疗大学）；Support Center for Students with Disabilities, Aoyama Gakuin University（残疾学生支持中心，上智大学）
Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English
HiTZ Center, University of the Basque Country (UPV/EHU)（巴斯克大学（UPV/EHU）HiTZ中心）；University of Alicante (UA)（阿利坎特大学（UA））；GOI Institute, Basque Summer University (UEU)（巴斯克夏季大学（UEU）GOI研究所）
