arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.14744cs.SDeess.AS
收藏

解锁大型音频-语言模型以实现交互式语言学习

Unlocking Large Audio-Language Models for Interactive Language Learning

National University of Singapore(新加坡国立大学)

Hongfu Liu, Zhouying Cui, Xiangming Gu, Ye Wang

AI总结:

本文提出通过指令微调ALMs,利用L2-Arctic-plus数据集提升交互式语言学习中的发音训练效果。

摘要:

实现母语以外的语言(L2)的发音熟练度仍是一个挑战,尽管已经开发了计算机辅助发音训练(CAPT)系统。传统的CAPT系统常常提供不直观的反馈,缺乏可操作的指导,限制了其有效性。最近在音频-语言模型(ALMs)方面的进展为这些系统提供了改进的潜力,通过提供更用户友好的反馈。在本工作中,我们通过引入包含详细错误解释和改进建议的英语数据集L2-Arctic-plus,研究了用于聊天式发音训练的ALMs。我们在此数据集上对级联ASR+LLMs和现有ALMs进行基准测试,特别是在检测发音错误和生成可操作反馈方面。为了提高性能,我们进一步提出在L2-Arctic-plus上对ALMs进行指令微调。实验结果表明,我们的指令微调模型在发音错误检测和建议生成方面,在客观和人工评估上都显著优于现有基线,突显了所提出数据集的价值。

英文摘要:

Achieving pronunciation proficiency in a second language (L2) remains a challenge, despite the development of Computer-Assisted Pronunciation Training (CAPT) systems. Traditional CAPT systems often provide unintuitive feedback that lacks actionable guidance, limiting its effectiveness. Recent advancements in audio-language models (ALMs) offer the potential to enhance these systems by providing more user-friendly feedback. In this work, we investigate ALMs for chat-based pronunciation training by introducing L2-Arctic-plus, an English dataset with detailed error explanations and actionable suggestions for improvement. We benchmark cascaded ASR+LLMs and existing ALMs on this dataset, specifically in detecting mispronunciation and generating actionable feedback. To improve the performance, we further propose to instruction-tune ALMs on L2-Arctic-plus. Experimental results demonstrate that our instruction-tuned models significantly outperform existing baselines on mispronunciation detection and suggestion generation in terms of both objective and human evaluation, highlighting the value of the proposed dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑