Ola：推动全模态语言模型的前沿发展
Ola: Pushing the Frontiers of Omni-Modal Language Model
AI总结：
Ola提出一种全模态语言模型，通过渐进式跨模态对齐训练和架构改进，在图像、视频、音频理解上超越开源全模态模型并接近专用模型水平。
摘要：
大语言模型的最新进展，尤其是GPT-4o问世之后，激发了人们对开发能够理解更多模态的全模态模型日益增长的兴趣。尽管已经出现了一些开源替代方案，但在性能上仍与专用单模态模型存在明显差距。本文提出Ola，一种全模态语言模型，在图像、视频和音频理解方面与专用同类模型相比取得了具有竞争力的表现，在很大程度上推动了全模态语言模型的前沿发展。我们对构建稳健全模态模型所必需的架构设计、数据整理和训练策略进行了全面探索。Ola通过对主流基线进行若干关键且有效的改进，融合了先进的视觉理解和音频识别能力。此外，我们重新思考了全模态训练中的模态间关系，强调以视频作为核心桥梁进行跨模态对齐，并提出了一种渐进式训练流程，从差异最明显的模态开始，逐步过渡到更紧密的模态对齐。大量实验表明，Ola在所有模态上均超越了现有的开源全模态大语言模型，同时在与同等规模的最先进专用模型相比时取得了极具竞争力的性能。我们致力于将Ola打造为完全开放的全模态理解解决方案，以推动这一新兴领域的未来研究。模型权重、代码和数据已在https://github.com/Ola-Omni/Ola开源。
英文摘要：
Recent advances in large language models, particularly following GPT-4o, have sparked increasing interest in developing omni-modal models capable of understanding more modalities. While some open-source alternatives have emerged, there is still a notable lag behind specialized single-modality models in performance. In this paper, we present Ola, an Omni-modal Language model that achieves competitive performance across image, video, and audio understanding compared to specialized counterparts, pushing the frontiers of the omni-modal language model to a large extent. We conduct a comprehensive exploration of architectural design, data curation, and training strategies essential for building a robust omni-modal model. Ola incorporates advanced visual understanding and audio recognition capabilities through several critical and effective improvements over mainstream baselines. Moreover, we rethink inter-modal relationships during omni-modal training, emphasizing cross-modal alignment with video as a central bridge, and propose a progressive training pipeline that begins with the most distinct modalities and gradually moves towards closer modality alignment. Extensive experiments demonstrate that Ola surpasses existing open omni-modal LLMs across all modalities while achieving highly competitive performance compared to state-of-the-art specialized models of similar sizes. We aim to make Ola a fully open omni-modal understanding solution to advance future research in this emerging field. Model weights, code, and data are open-sourced at https://github.com/Ola-Omni/Ola.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
