基于多模态大语言模型的发音评估
Pronunciation Assessment with Multi-modal Large Language Models
AI总结：
本文提出一种基于大语言模型的发音评分系统，利用语音编码器和适配器对齐特征，结合提示文本预测准确性与流利度，在Speechocean762数据集上取得有竞争力结果。
摘要：
大型语言模型（LLMs）以其强大的对话能力而闻名，被广泛认为是教育领域的杰出工具，特别是在语言学习的自动化智能教学系统中。在本文中，我们提出了一种基于LLMs的评分系统，其动机源于LLMs在文本相关评分任务中的积极影响。具体来说，语音编码器首先将学习者的语音映射为上下文特征。然后，适配器层将这些特征转换，以与潜在空间中的文本嵌入对齐。评估任务特定的前缀和提示文本被嵌入，并与模态适配器层生成的特征拼接，使LLMs能够预测准确性和流利度分数。我们的实验表明，所提出的评分系统在Speechocean762数据集上与基线相比取得了有竞争力的结果。此外，我们还进行了消融研究，以更好地理解提示文本和训练策略在所提出的评分系统中的作用。
英文摘要：
Large language models (LLMs), renowned for their powerful conversational abilities, are widely recognized as exceptional tools in the field of education, particularly in the context of automated intelligent instruction systems for language learning. In this paper, we propose a scoring system based on LLMs, motivated by their positive impact on text-related scoring tasks. Specifically, the speech encoder first maps the learner's speech into contextual features. The adapter layer then transforms these features to align with the text embedding in latent space. The assessment task-specific prefix and prompt text are embedded and concatenated with the features generated by the modality adapter layer, enabling the LLMs to predict accuracy and fluency scores. Our experiments demonstrate that the proposed scoring systems achieve competitive results compared to the baselines on the Speechocean762 datasets. Moreover, we also conducted an ablation study to better understand the contributions of the prompt text and training strategy in the proposed scoring system.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
