arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.04718cs.CL
收藏

评估文本风格迁移评估:是否存在可靠的指标?

Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?

Charles University(查理大学)

Sourabrata Mukherjee, Atul Kr. Ojha, John P. McCrae, Ondrej Dusek

AI总结:

本文系统评估了多种自动指标在文本风格迁移任务中的有效性,发现新应用的高级NLP指标和基于大语言模型的评估优于现有指标,集成方法更具潜力。

摘要:

文本风格迁移(TST)是将文本转换为反映特定风格同时保留其原始内容的任务。评估TST输出是一个多维挑战,需要评估风格迁移准确性、内容保留和自然度。使用人工评估是理想的但成本高昂,这在自然语言处理(NLP)任务中很常见,然而,TST的自动指标并未像机器翻译或摘要等任务的指标那样受到同等关注。在本文中,我们考察了来自更广泛NLP任务的现有和新颖指标,用于TST评估,重点关注两个流行的子任务:情感迁移和去毒化,并在包含英语、印地语和孟加拉语的多语言环境中进行。通过进行与人类判断相关的元评估,我们展示了这些指标在单独使用和集成使用时的有效性。此外,我们调查了大型语言模型(LLMs)作为TST评估工具的潜力。我们的发现强调,新应用的高级NLP指标和基于LLM的评估比现有TST指标提供了更好的见解。我们的预言机集成方法显示出更大的潜力。

英文摘要:

Text style transfer (TST) is the task of transforming a text to reflect a particular style while preserving its original content. Evaluating TST outputs is a multidimensional challenge, requiring the assessment of style transfer accuracy, content preservation, and naturalness. Using human evaluation is ideal but costly, as is common in other natural language processing (NLP) tasks, however, automatic metrics for TST have not received as much attention as metrics for, e.g., machine translation or summarization. In this paper, we examine both set of existing and novel metrics from broader NLP tasks for TST evaluation, focusing on two popular subtasks, sentiment transfer and detoxification, in a multilingual context comprising English, Hindi, and Bengali. By conducting meta-evaluation through correlation with human judgments, we demonstrate the effectiveness of these metrics when used individually and in ensembles. Additionally, we investigate the potential of large language models (LLMs) as tools for TST evaluation. Our findings highlight newly applied advanced NLP metrics and LLM-based evaluations provide better insights than existing TST metrics. Our oracle ensemble approaches show even more potential.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑