Large language models (LLMs) have revolutionized NLP research. Notably, in-context learning enables their use as evaluation metrics for natural language generation, making them particularly advantageous in low-resource scenarios and time-restricted applications. In this work, we introduce PrExMe, a large-scale Prompt Exploration for Metrics, where we evaluate more than 720 prompt templates for open-source LLM-based metrics on machine translation (MT) and summarization datasets, totalling over 6.6M evaluations. This extensive comparison (1) benchmarks recent open-source LLMs as metrics and (2) explores the stability and variability of different prompting strategies. We discover that, on the one hand, there are scenarios for which prompts are stable. For instance, some LLMs show idiosyncratic preferences and favor to grade generated texts with textual labels while others prefer to return numeric scores. On the other hand, the stability of prompts and model rankings can be susceptible to seemingly innocuous changes. For example, changing the requested output format from "0 to 100" to "-1 to +1" can strongly affect the rankings in our evaluation. Our study contributes to understanding the impact of different prompting approaches on LLM-based metrics for MT and summarization evaluation, highlighting the most stable prompting patterns and potential limitations.
Don't Believe Everything You Read: Enhancing Summarization Interpretability through Automatic Identification of Hallucinations in Large Language Models
Unifying Cross-lingual Summarization and Machine Translation with Compression Rate
Beijing Institute of Technology（北京理工大学）；Southeast Academy of Information Technology（东南信息技术研究院）；Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications（北京海量语言信息处理与云计算应用工程研究中心）
Language on Demand, Knowledge at Core: Composing LLMs with Encoder-Decoder Translation Models for Extensible Multilinguality
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences（智能信息处理重点实验室，计算技术研究所，中国科学院）；State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）
Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models
Southeast University（东南大学）；Nanjing University（南京大学）；The University of Sydney（悉尼大学）；Shenzhen Research Institute（深圳研究院）；Microsoft（微软公司）；Nanyang Technological University（南洋理工大学）
ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generation
School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；NiuTrans Research（NiuTrans研究院）；Institute of Psychology, CAS（中国科学院心理研究所）
GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models
University of Helsinki（赫尔辛基大学）；Technical University of Darmstadt（达姆施塔特技术大学）；University of Munich（慕尼黑大学）；Nanjing University（南京大学）；ELLIS Institute Finland（芬兰ELLIS研究所）；University of Turku（图尔库大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages
Sharif University of Technology（谢里夫理工学院）；Vellore Institute of Technology（韦洛雷理工学院）；IIT Kharagpur（印度理工学院达卡分校）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding
Hasso Plattner Institute for Digital Engineering, University of Potsdam（波茨坦大学哈索·普拉特纳数字工程研究所）；Qualcomm AI Research（高通人工智能研究）；University of Amsterdam（阿姆斯特丹大学）
The Cross-lingual Conversation Summarization Challenge
Westlake University（西湖大学）；Sichuan Lan-bridge Information Technology Co., Ltd.（四川蓝桥信息技术有限公司）；Zhejiang University（浙江大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Michigan, Ann Arbor（密歇根大学安娜堡分校）
Bidirectional Language Models Are Also Few-shot Learners
University of Pennsylvania（宾夕法尼亚大学）；Microsoft（微软）；Google Research（谷歌研究院）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）
大语言模型，例如GPT-3（Brown等人，2020）可以执行任意任务，而无需在仅使用几个标记示例进行提示后进行微调。一个任意的任务可以被重新表述为一个自然语言提示，一个语言模型可以被要求生成完成，间接地执行任务的范式称为基于语义的学习。到目前为止，基于突现的学习能力主要是针对单向语言模型来证明的。然而，在去噪目标上预训练的双向语言模型（例如掩蔽语言建模）为迁移学习产生更强的学习表示。这激发了促进双向模式的可能性，但它们的预培训目标使它们在很大程度上与现有的激励范式不相容。我们提出了SAP（顺序自回归拟合），一种技术，使双向模型的提示。利用机器翻译任务作为案例研究，我们提出了双向mT5模型（Xue et al.，2021）与SAP进行比较，并证明其Few-Shot和zero-shot转换优于单向模型（如GPT-3和XGLM）的Few-Shot转换（Lin et al.，2021），尽管mT5的参数减少了约50%。我们进一步表明SAP在问答和摘要方面是有效的。这是第一次，我们的研究结果表明，基于语义的学习是一个更广泛的一类语言模型，而不仅仅是单向模型的新兴属性。
Evaluating the Efficacy of Length-Controllable Machine Translation
Academy for Advanced Interdisciplinary Studies, Peking University（北京大学前沿交叉学科研究院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Artificial Intelligence Application Research Center, Huawei Technologies（华为技术有限公司人工智能应用研究中心）；School of Mathematical Sciences, Peking University（北京大学数学科学学院）
Hierarchical Learning for Generation with Long Source Sequences
Birch AI（桦树人工智能公司）；The University of Chicago（芝加哥大学）；Toyota Technological Institute at Chicago（芝加哥丰田理工学院）；Paul G. Allen School of Computer Science & Engineering（保罗·G·艾伦计算机科学与工程学院）；University of Washington（华盛顿大学）
