BeAts：使用多模态注意力融合的孟加拉语言语行为识别
BeAts: Bengali Speech Acts Recognition using Multimodal Attention Fusion
AI总结：
本文提出 BeAts，融合 wav2vec2.0 音频表征与 MarianMT 文本翻译，并通过多模态注意力识别孟加拉语言语行为，性能显著优于单模态和简单双模态基线。
摘要：
口语通常利用语调、节奏、强度和结构来传达意图，而意图可能会因其话语的言语节奏不同而得到不同解释。这些言语行为构成了交流的基础，并且在表达上具有语言独特性。基于注意力的模型近期取得进展，展现出从多语言数据集中学习强大表征的能力，已在语音任务中表现良好，因此非常适合为低资源语言中的特定任务建模。本文中，我们开发了一种新颖的多模态方法，结合两个模型，即用于音频的 wav2vec2.0 和用于文本翻译的 MarianMT，并通过多模态注意力融合，在我们准备的孟加拉语语音语料上预测言语行为。我们还表明，我们的模型 BeAts（Bengali speech acts recognition using Multimodal Attention Fusion，使用多模态注意力融合的孟加拉语言语行为识别）显著优于仅使用语音数据的单模态基线，以及同时使用语音和文本数据的较简单双模态融合方法。项目页面：https://soumitri2001.github.io/BeAts
英文摘要：
Spoken languages often utilise intonation, rhythm, intensity, and structure, to communicate intention, which can be interpreted differently depending on the rhythm of speech of their utterance. These speech acts provide the foundation of communication and are unique in expression to the language. Recent advancements in attention-based models, demonstrating their ability to learn powerful representations from multilingual datasets, have performed well in speech tasks and are ideal to model specific tasks in low resource languages. Here, we develop a novel multimodal approach combining two models, wav2vec2.0 for audio and MarianMT for text translation, by using multimodal attention fusion to predict speech acts in our prepared Bengali speech corpus. We also show that our model BeAts ($\underline{\textbf{Be}}$ngali speech acts recognition using Multimodal $\underline{\textbf{At}}$tention Fu$\underline{\textbf{s}}$ion) significantly outperforms both the unimodal baseline using only speech data and a simpler bimodal fusion using both speech and text data. Project page: https://soumitri2001.github.io/BeAts
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
