Efficiency is a key property to foster inclusiveness and reduce environmental costs, especially in an era of LLMs. In this work, we provide a comprehensive evaluation of efficiency for MT evaluation metrics. Our approach involves replacing computation-intensive transformers with lighter alternatives and employing linear and quadratic approximations for alignment algorithms on top of LLM representations. We evaluate six (reference-free and reference-based) metrics across three MT datasets and examine 16 lightweight transformers. In addition, we look into the training efficiency of metrics like COMET by utilizing adapters. Our results indicate that (a) TinyBERT provides the optimal balance between quality and efficiency, (b) CPU speed-ups are more substantial than those on GPU; (c) WMD approximations yield no efficiency gains while reducing quality and (d) adapters enhance training efficiency (regarding backward pass speed and memory requirements) as well as, in some cases, metric quality. These findings can help to strike a balance between evaluation speed and quality, which is essential for effective NLG systems. Furthermore, our research contributes to the ongoing efforts to optimize NLG evaluation metrics with minimal impact on performance. To our knowledge, ours is the most comprehensive analysis of different aspects of efficiency for MT metrics conducted so far.
One Stone, Four Birds: A Comprehensive Solution for QA System Using Supervised Contrastive Learning
Graduate School of Information Science and Electrical Engineering, Kyushu University(九州大学信息科学与电气工程研究院);Faculty of Information Science and Electrical Engineering, Kyushu University(九州大学信息科学与电气工程学部)
Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis
School of Computer Science and Engineering, The Hebrew University of Jerusalem(希伯来大学耶路撒冷分校计算机科学与工程学院);School of Computing and Information Systems, University of Melbourne(墨尔本大学计算与信息学院)
Indian Institute of Technology, Madras(马德拉斯印度理工学院);AI4Bharat;National Institute of Information and Communications Technology(国立信息与通信技术研究所);Microsoft(微软)
Impact of enriched meaning representations for language generation in dialogue tasks: A comprehensive exploration of the relevance of tasks, corpora and metrics
University of the Basque Country (UPV/EHU)(巴斯克大学(UPV/EHU))
Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutions
Peking University(北京大学);Chongqing Big Data Research Institute, Peking University(重庆大数据研究院(北京大学));Beijing International Center for Mathematical Research, Peking University(北京国际数学研究中心(北京大学))
SGSimEval: A Comprehensive Multifaceted and Similarity-Enhanced Benchmark for Automatic Survey Generation Systems
The Hong Kong Polytechnic University(香港理工大学);The Education University of Hong Kong(香港教育大学);China Mobile (Hong Kong) Innovation and Research Institute(中国移动(香港)创新与研究院);Lingnan University(岭南大学)
SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version
FPT University(FPT大学);Aalborg University(奥尔堡大学);Technische Universität Berlin(柏林技术大学);RMIT University(皇家理工大学);German Research Center for Artificial Intelligence(德国人工智能研究中心);HiveIntel GmbH(HiveIntel公司)