arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.09593cs.CLcs.LG
收藏

EffEval:机器翻译评估指标效率的综合评估

EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metrics

Bielefeld University(比勒费尔德大学);Technical University of Darmstadt(达姆施塔特工业大学)

Daniil Larionov, Jens Grünwald, Christoph Leiter, Steffen Eger

AI总结:

本文提出 EffEval 框架,通过轻量级 transformer、对齐算法近似和适配器等方法,系统评估机器翻译评估指标在推理与训练中的效率与质量权衡,为高效 NLG 评估提供依据。

摘要:

效率是促进包容性和降低环境成本的关键属性,尤其是在大语言模型(LLM)时代。在本工作中,我们对机器翻译(MT)评估指标的效率进行了全面评估。我们的方法包括用更轻量的替代方案替换计算密集型的 transformer,并在 LLM 表征之上对对齐算法采用线性和二次近似。我们在三个 MT 数据集上评估了六种(无参考和基于参考的)指标,并考察了 16 种轻量级 transformer。此外,我们还通过使用适配器(adapters)研究了 COMET 等指标的训练效率。我们的结果表明:(a)TinyBERT 在质量和效率之间提供了最优平衡;(b)CPU 上的加速比 GPU 上更为显著;(c)WMD 近似没有带来效率提升,同时降低了质量;(d)适配器提高了训练效率(就反向传播速度和内存需求而言),并且在某些情况下还提高了指标质量。这些发现有助于在评估速度和质量之间取得平衡,这对于有效的自然语言生成(NLG)系统至关重要。此外,我们的研究为持续优化 NLG 评估指标并以最小性能影响为目标的努力做出了贡献。据我们所知,我们的工作是迄今为止对 MT 指标效率的不同方面所进行的最全面的分析。

英文摘要:

Efficiency is a key property to foster inclusiveness and reduce environmental costs, especially in an era of LLMs. In this work, we provide a comprehensive evaluation of efficiency for MT evaluation metrics. Our approach involves replacing computation-intensive transformers with lighter alternatives and employing linear and quadratic approximations for alignment algorithms on top of LLM representations. We evaluate six (reference-free and reference-based) metrics across three MT datasets and examine 16 lightweight transformers. In addition, we look into the training efficiency of metrics like COMET by utilizing adapters. Our results indicate that (a) TinyBERT provides the optimal balance between quality and efficiency, (b) CPU speed-ups are more substantial than those on GPU; (c) WMD approximations yield no efficiency gains while reducing quality and (d) adapters enhance training efficiency (regarding backward pass speed and memory requirements) as well as, in some cases, metric quality. These findings can help to strike a balance between evaluation speed and quality, which is essential for effective NLG systems. Furthermore, our research contributes to the ongoing efforts to optimize NLG evaluation metrics with minimal impact on performance. To our knowledge, ours is the most comprehensive analysis of different aspects of efficiency for MT metrics conducted so far.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑