arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.14224cs.CL
收藏

mmT5:模块化多语言预训练解决源语言幻觉

mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinations

Google DeepMind(谷歌DeepMind)

Jonas Pfeiffer, Francesco Piccinno, Massimo Nicosia, Xinyi Wang, Machel Reid, Sebastian Ruder

AI总结:

针对多语言模型在少样本下生成错误语言的问题,提出模块化模型mmT5,通过语言特定模块分离信息并解决微调表示漂移,在40多种语言任务上大幅超越mT5,将零样本正确语言生成率从7%提升至99%。

摘要:

随着语言覆盖范围的增加,多语言序列到序列模型表现不佳,且在少样本设置下无法一致地以正确的目标语言生成文本。为了应对这些挑战,我们提出了mmT5,一个模块化的多语言序列到序列模型。mmT5在预训练期间利用语言特定模块,将语言特定信息与语言无关信息分离开来。我们发现微调期间的表示漂移是模块化生成模型的一个关键限制,并开发了能够实现有效零样本迁移的策略。在40多种语言的代表性自然语言理解和生成任务上,我们的模型在相同参数规模下大幅优于mT5。与mT5相比,mmT5将零样本设置下以正确语言生成文本的比率从7%提高到99%,从而极大地缓解了源语言幻觉问题。

英文摘要:

Multilingual sequence-to-sequence models perform poorly with increased language coverage and fail to consistently generate text in the correct target language in few-shot settings. To address these challenges, we propose mmT5, a modular multilingual sequence-to-sequence model. mmT5 utilizes language-specific modules during pre-training, which disentangle language-specific information from language-agnostic information. We identify representation drift during fine-tuning as a key limitation of modular generative models and develop strategies that enable effective zero-shot transfer. Our model outperforms mT5 at the same parameter sizes by a large margin on representative natural language understanding and generation tasks in 40+ languages. Compared to mT5, mmT5 raises the rate of generating text in the correct language under zero-shot settings from 7% to 99%, thereby greatly alleviating the source language hallucination problem.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑