OneLLM：一个用语言对齐所有模态的框架
OneLLM: One Framework to Align All Modalities with Language
AI总结：
本文提出 OneLLM 统一框架，通过通用投影模块和渐进对齐将八种模态接入 LLM，并构建 200 万条多模态指令数据，在 25 个基准上取得优异表现。
摘要：
多模态大语言模型（MLLM）因其强大的多模态理解能力而受到广泛关注。然而，现有工作严重依赖特定于模态的编码器，这些编码器通常在架构上各不相同，并且仅限于常见模态。本文提出 OneLLM，这是一种通过统一框架将八种模态与语言对齐的 MLLM。我们通过一个统一多模态编码器和渐进式多模态对齐流程实现这一点。具体而言，我们首先训练一个图像投影模块，以连接视觉编码器与 LLM。随后，我们通过混合多个图像投影模块并采用动态路由，构建通用投影模块（UPM）。最后，我们使用 UPM 渐进地将更多模态与 LLM 对齐。为充分释放 OneLLM 在指令遵循方面的潜力，我们还构建了一个全面的多模态指令数据集，包含来自图像、音频、视频、点云、深度/法线图、IMU 和 fMRI 脑活动的 200 万条数据。OneLLM 在 25 个多样化基准上接受评估，涵盖多模态描述、问答和推理等任务，并取得优异性能。代码、数据、模型和在线演示可在 https://github.com/csuhan/OneLLM 获取。
英文摘要：
Multimodal large language models (MLLMs) have gained significant attention due to their strong multimodal understanding capability. However, existing works rely heavily on modality-specific encoders, which usually differ in architecture and are limited to common modalities. In this paper, we present OneLLM, an MLLM that aligns eight modalities to language using a unified framework. We achieve this through a unified multimodal encoder and a progressive multimodal alignment pipeline. In detail, we first train an image projection module to connect a vision encoder with LLM. Then, we build a universal projection module (UPM) by mixing multiple image projection modules and dynamic routing. Finally, we progressively align more modalities to LLM with the UPM. To fully leverage the potential of OneLLM in following instructions, we also curated a comprehensive multimodal instruction dataset, including 2M items from image, audio, video, point cloud, depth/normal map, IMU and fMRI brain activity. OneLLM is evaluated on 25 diverse benchmarks, encompassing tasks such as multimodal captioning, question answering and reasoning, where it delivers excellent performance. Code, data, model and online demo are available at https://github.com/csuhan/OneLLM
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
