arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.09512cs.CLcs.LG
收藏

PIER:一种评估语码转换中关键内容的新指标

PIER: A Novel Metric for Evaluating What Matters in Code-Switching

Karlsruhe Institut of Technology (KIT)(卡尔斯鲁厄理工学院)

Enes Yavuz Ugan, Ngoc-Quan Pham, Leonard Bärmann, Alex Waibel

AI总结:

针对语码转换任务,提出PIER指标,仅关注兴趣词以更准确评估性能,揭示现有指标高估效果,未来改进空间大。

摘要:

语码转换,即在同一话语中交替使用多种语言,对自动语音识别构成了重大挑战。尽管该任务具有独特性,但性能通常使用诸如词错误率(WER)等既有指标来衡量。然而,在本文中,我们质疑这些通用指标是否能准确评估语码转换的性能。具体而言,我们使用连接时序分类(CTC)模型和编码器-解码器模型,展示了在来自矩阵语言和嵌入语言的非语码转换数据上进行微调,虽然能改善语码转换测试集上的经典指标,但实际的语码转换词汇性能却恶化(如预期)。因此,我们提出了兴趣点错误率(PIER),这是WER的一种变体,仅关注特定的兴趣词汇。我们在语码转换话语上实例化PIER,并表明该指标能更准确地描述语码转换性能,显示出未来工作存在巨大的改进空间。这种聚焦评估能够更精确地评估模型性能,特别是在词间和词内语码转换等具有挑战性的方面。

英文摘要:

Code-switching, the alternation of languages within a single discourse, presents a significant challenge for Automatic Speech Recognition. Despite the unique nature of the task, performance is commonly measured with established metrics such as Word-Error-Rate (WER). However, in this paper, we question whether these general metrics accurately assess performance on code-switching. Specifically, using both Connectionist-Temporal-Classification and Encoder-Decoder models, we show fine-tuning on non-code-switched data from both matrix and embedded language improves classical metrics on code-switching test sets, although actual code-switched words worsen (as expected). Therefore, we propose Point-of-Interest Error Rate (PIER), a variant of WER that focuses only on specific words of interest. We instantiate PIER on code-switched utterances and show that this more accurately describes the code-switching performance, showing huge room for improvement in future work. This focused evaluation allows for a more precise assessment of model performance, particularly in challenging aspects such as inter-word and intra-word code-switching.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑