基于翻译对齐的视觉与语言模型参数高效跨语言迁移
Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment
AI总结：
本文提出基于翻译对齐的参数高效跨语言迁移框架，在11种语言、三种学习场景下显著缩小多语言性能差距，仅需极少量额外参数即可提升低资源跨语言迁移效果。
摘要：
预训练的视觉与语言模型（如CLIP）在连接图像与文本方面取得了显著成功，但其主要关注英语文本。尽管近期已有工作尝试将CLIP扩展到其他语言，但由于资源分布不均，不同语言之间的性能差异仍然存在。此外，现有的跨语言迁移方法在处理大量语言时会消耗过多资源。为此，我们提出了一种新的参数高效跨语言迁移学习框架，该框架利用基于翻译的对齐方法来缓解多语言差异，并探索了参数高效微调方法以实现参数高效的跨语言迁移。在XTD和Multi30K数据集上，覆盖11种语言，在零样本、少样本和全数据集学习场景下的大量实验表明，我们的框架显著减少了语言之间的多语言差异，并改善了跨语言迁移结果，尤其是在低资源场景下，同时与完整模型相比仅保留和微调极少量的参数（例如，在少样本学习场景下，我们的框架每种语言仅需完整模型0.16%的额外参数）。代码可在\url{https://github.com/eric-ai-lab/PECTVLM}获取。
英文摘要：
Pre-trained vision and language models such as CLIP have witnessed remarkable success in connecting images and texts with a primary focus on English texts. Despite recent efforts to extend CLIP to support other languages, disparities in performance among different languages have been observed due to uneven resource availability. Additionally, current cross-lingual transfer methods of those pre-trained models would consume excessive resources for a large number of languages. Therefore, we propose a new parameter-efficient cross-lingual transfer learning framework that utilizes a translation-based alignment method to mitigate multilingual disparities and explores parameter-efficient fine-tuning methods for parameter-efficient cross-lingual transfer. Extensive experiments on XTD and Multi30K datasets, covering 11 languages under zero-shot, few-shot, and full-dataset learning scenarios, show that our framework significantly reduces the multilingual disparities among languages and improves cross-lingual transfer results, especially in low-resource scenarios, while only keeping and fine-tuning an extremely small number of parameters compared to the full model (e.g., Our framework only requires 0.16\% additional parameters of a full-model for each language in the few-shot learning scenario). The codes are available at \url{https://github.com/eric-ai-lab/PECTVLM}. The codes are available at \url{https://github.com/eric-ai-lab/PECTVLM}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
