arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.17643cs.CLcs.AI
收藏

Tonguescape:探索语言模型对元音发音的理解

Tonguescape: Exploring Language Models Understanding of Vowel Articulation

Nara Institute of Science and Technology (NAIST)(奈良科学技术研究所(NAIST))

Haruki Sakajo, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

AI总结:

本研究基于实时MRI构建数据集,探究语言模型能否利用视觉信息将真实舌头位置与元音发音关联,发现提供参考示例时模型具备理解潜力,否则存在困难。

摘要:

元音主要由舌头位置决定。人类通过自身经验以及使用MRI等明确的客观观察,发现了元音发音的这些特征。借助这些知识和经验,我们能够解释和理解舌头位置与元音之间的关系,这有助于语言学习者学习发音。由于语言模型(LMs)在包含语言学和医学领域的大量数据上进行训练,我们的初步研究表明,LM能够解释元音的发音机制。然而,目前尚不清楚多模态LM(如视觉LM)是否能将文本信息与视觉信息对齐。由此产生了一个问题:LM是否将真实的舌头位置与元音发音联系起来?在本研究中,我们从现有的实时MRI数据集创建了视频和图像数据集,并利用基于视觉的信息,探讨了LM是否能够基于舌头位置理解元音发音。研究结果表明,当提供参考示例时,LM展现出理解元音和舌头位置的潜力,但在没有参考示例的情况下则存在困难。我们在GitHub上提供了数据集构建的代码。

英文摘要:

Vowels are primarily characterized by tongue position. Humans have discovered these features of vowel articulation through their own experience and explicit objective observation such as using MRI. With this knowledge and our experience, we can explain and understand the relationship between tongue positions and vowels, and this knowledge is helpful for language learners to learn pronunciation. Since language models (LMs) are trained on a large amount of data that includes linguistic and medical fields, our preliminary studies indicate that an LM is able to explain the pronunciation mechanisms of vowels. However, it is unclear whether multi-modal LMs, such as vision LMs, align textual information with visual information. One question arises: do LMs associate real tongue positions with vowel articulation? In this study, we created video and image datasets from the existing real-time MRI dataset and investigated whether LMs can understand vowel articulation based on tongue positions using vision-based information. Our findings suggest that LMs exhibit potential for understanding vowels and tongue positions when reference examples are provided while they have difficulties without them. Our code for dataset building is available on GitHub.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑