arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.06648cs.CLcs.AIcs.LG
收藏

SignBLEU:多通道手语翻译的自动评估

SignBLEU: Automatic Evaluation of Multi-channel Sign Language Translation

EQ4ALL

Jung-Ho Kim, Mathew Huerta-Enochian, Changyong Ko, Du Hui Lee

AI总结:

本文提出多通道手语翻译任务和SignBLEU指标,以同时评估手动手动与非手动信号,并在多个语料库上验证其与人类判断的相关性优于现有指标。

摘要:

手语是一种多通道语言,不仅通过手部(手动信号)传递信息,还通过面部表情和上半身动作(非手动信号)传递信息。然而,由于自动手语翻译通常通过生成单一注释序列来完成,研究者往往舍弃非手动信号和同时出现的手动信号,而采用简化后的手动注释列表。这可能导致严重的信息丢失和歧义。本文引入了一个名为多通道手语翻译(MCSLT)的新任务,并提出了一种新的度量指标 SignBLEU,旨在捕捉多个信号通道。我们使用三个具有不同语言结构和转写方法的手语语料库,在一个系统级任务上验证了 SignBLEU,并通过两个片段级任务考察了其与人类判断的相关性。我们发现,与竞争指标相比,SignBLEU 始终与人类判断具有更好的相关性。为促进进一步的 MCSLT 研究,我们报告了三个手语语料库的基准分数,并在 https://github.com/eq4all-projects/SignBLEU 发布了 SignBLEU 的源代码。

英文摘要:

Sign languages are multi-channel languages that communicate information through not just the hands (manual signals) but also facial expressions and upper body movements (non-manual signals). However, since automatic sign language translation is usually performed by generating a single sequence of glosses, researchers eschew non-manual and co-occurring manual signals in favor of a simplified list of manual glosses. This can lead to significant information loss and ambiguity. In this paper, we introduce a new task named multi-channel sign language translation (MCSLT) and present a novel metric, SignBLEU, designed to capture multiple signal channels. We validated SignBLEU on a system-level task using three sign language corpora with varied linguistic structures and transcription methodologies and examined its correlation with human judgment through two segment-level tasks. We found that SignBLEU consistently correlates better with human judgment than competing metrics. To facilitate further MCSLT research, we report benchmark scores for the three sign language corpora and release the source code for SignBLEU at https://github.com/eq4all-projects/SignBLEU.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑