低秩与稀疏模型合并用于多语言语音识别与翻译
Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translation
AI总结：
针对多语言语音识别与翻译中的高计算成本、语言干扰等问题，提出低秩与稀疏模型合并方法LoRS-Merging，在10种语言上性能提升超20%，有效补充传统多语言训练策略。
摘要：
语言多样性对语音到文本（S2T）任务（如自动语音识别和翻译）构成了重大挑战。传统的多语言多任务训练方法旨在通过联合优化多种语言的多项语音识别和翻译任务来解决这一问题。尽管基于这些策略构建的模型（如Whisper）表现出强大的性能，但它们仍面临高计算成本、语言干扰、次优的训练配置以及可扩展性有限等问题。为了克服这些挑战，我们引入了LoRS-Merging（低秩与稀疏模型合并），这是一种新颖的技术，旨在高效整合在不同语言或任务上训练的模型，同时保持性能并降低计算开销。LoRS-Merging结合了低秩与稀疏剪枝，以保留必要结构，同时消除冗余参数，减轻语言干扰，并增强可扩展性。在10种语言上的实验结果表明，LoRS-Merging显著优于多语言多任务训练、顺序训练及其他合并方法，在归一化性能上实现了超过20%的提升。我们的研究结果表明，模型合并，特别是LoRS-Merging，是传统多语言训练策略在S2T应用中的一种可扩展且有效的补充。
英文摘要：
Language diversity presents a significant challenge in speech-to-text (S2T) tasks, such as automatic speech recognition and translation. Traditional multi-lingual multi-task training approaches aim to address this by jointly optimising multiple speech recognition and translation tasks across various languages. While models like Whisper, built on these strategies, demonstrate strong performance, they still face issues of high computational cost, language interference, suboptimal training configurations, and limited extensibility. To overcome these challenges, we introduce LoRS-Merging (low-rank and sparse model merging), a novel technique designed to efficiently integrate models trained on different languages or tasks while preserving performance and reducing computational overhead. LoRS-Merging combines low-rank and sparse pruning to retain essential structures while eliminating redundant parameters, mitigating language interference, and enhancing extensibility. Experimental results across 10 languages demonstrate that LoRS-Merging significantly outperforms multi-lingual multi-task training, sequential training, and other merging methods, achieving over 20% improvement in normalised performance. Our findings suggest that model merging, particularly LoRS-Merging, is a scalable and effective complement to traditional multi-lingual training strategies for S2T applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
