arXiv 2505.087512025-05-13cs.CLcs.CVcs.LG

Aya Vision：推进多语言多模态的前沿

Aya Vision: Advancing the Frontier of Multilingual Multimodality

Cohere Labs；Cohere

Saurabh Dash, Yiyang Nan, John Dang, Arash Ahmadian, Shivalika Singh, Madeline Smith, Bharat Venkitesh, Vlad Shmyhlo, Viraat Aryabumi, Walter Beller-Morales, Je… 展开作者

Saurabh Dash, Yiyang Nan, John Dang, Arash Ahmadian, Shivalika Singh, Madeline Smith, Bharat Venkitesh, Vlad Shmyhlo, Viraat Aryabumi, Walter Beller-Morales, Jeremy Pekmez, Jason Ozuzu, Pierre Richemond, Acyr Locatelli, Nick Frosst, Phil Blunsom, Aidan Gomez, Ivan Zhang, Marzieh Fadaee, Manoj Govindassamy, Sudip Roy, Matthias Gallé, Beyza Ermis, Ahmet Üstün, Sara Hooker

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多语言多模态建模中的数据稀缺、翻译失真与灾难性遗忘问题，本文提出合成标注框架和跨模态模型合并技术，打造 Aya Vision 8B/32B 并以更小规模取得领先性能。

摘要：

构建多模态语言模型从根本上具有挑战性：它需要对齐视觉与语言模态，整理高质量指令数据，并避免在引入视觉后现有纯文本能力发生退化。这些困难在多语言场景中进一步放大：不同语言对多模态数据的需求加剧了既有数据稀缺，机器翻译常常扭曲语义，而且灾难性遗忘更加明显。为应对上述挑战，我们提出了贯穿数据与建模的新技术。首先，我们开发了一个合成标注框架，用于整理高质量、多样化的多语言多模态指令数据，使 Aya Vision 模型能够针对多种语言的多模态输入生成自然且更受人类偏好的回答。作为补充，我们提出一种跨模态模型合并技术，以缓解灾难性遗忘，在有效保留纯文本能力的同时增强多模态生成性能。与 Qwen-2.5-VL-7B、Pixtral-12B，甚至大得多的 Llama-3.2-90B-Vision 等强大多模态模型相比，Aya-Vision-8B 取得了同级别最佳性能。我们进一步用 Aya-Vision-32B 扩展该方法，其表现优于 Molmo-72B 和 LLaMA-3.2-90B-Vision 等规模超过其两倍的模型。我们的工作推进了多模态前沿上的多语言进展，并为有效降低算力需求同时提供极高性能的技术提供了洞见。

英文摘要：

Building multimodal language models is fundamentally challenging: it requires aligning vision and language modalities, curating high-quality instruction data, and avoiding the degradation of existing text-only capabilities once vision is introduced. These difficulties are further magnified in the multilingual setting, where the need for multimodal data in different languages exacerbates existing data scarcity, machine translation often distorts meaning, and catastrophic forgetting is more pronounced. To address the aforementioned challenges, we introduce novel techniques spanning both data and modeling. First, we develop a synthetic annotation framework that curates high-quality, diverse multilingual multimodal instruction data, enabling Aya Vision models to produce natural, human-preferred responses to multimodal inputs across many languages. Complementing this, we propose a cross-modal model merging technique that mitigates catastrophic forgetting, effectively preserving text-only capabilities while simultaneously enhancing multimodal generative performance. Aya-Vision-8B achieves best-in-class performance compared to strong multimodal models such as Qwen-2.5-VL-7B, Pixtral-12B, and even much larger Llama-3.2-90B-Vision. We further scale this approach with Aya-Vision-32B, which outperforms models more than twice its size, such as Molmo-72B and LLaMA-3.2-90B-Vision. Our work advances multilingual progress on the multi-modal frontier, and provides insights into techniques that effectively bend the need for compute while delivering extremely high performance.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。