OmniBind:通过绑定空间实现大规模全模态多模态表示
OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces
AI总结:
OmniBind通过绑定多个预训练专家模型的空间,构建了70亿至300亿参数的大规模全模态表示模型,支持3D、音频、图像和语言输入,训练高效且性能优越。
摘要:
近年来,涉及多种模态的人机交互展现出了广阔的应用前景,例如GPT-4o和Gemini。鉴于多模态联合表示在理解和生成流程中的基础性作用,高质量的全模态联合表示将是朝着协同处理更多样化多模态信息迈出的一步。在这项工作中,我们提出了OmniBind,这是一种大规模多模态联合表示模型,参数规模从70亿到300亿不等,支持3D、音频、图像和语言输入。由于所有模态之间的数据对稀缺,我们没有从头开始训练大型模型,而是提出将各种预训练专家模型的空间重新映射并绑定在一起。这种方法通过间接增加模型参数和已见数据量来实现“扩展”。为了有效整合各种空间,我们通过学习具有两个目标的路由器来动态分配不同空间的权重:跨模态整体对齐和语言表示解耦。值得注意的是,由于绑定和路由空间都只需要轻量级网络,OmniBind的训练效率极高。学习最大的300亿参数模型仅需要不成对的单模态数据,并且在单个8-4090节点上大约需要3天。大量实验证明了OmniBind作为全模态表示模型的多功能性和优越性,突显了其在各种应用中的巨大潜力,例如任意查询和可组合的多模态理解。
英文摘要:
Recently, human-computer interaction with various modalities has shown promising applications, like GPT-4o and Gemini. Given the foundational role of multimodal joint representation in understanding and generation pipelines, high-quality omni joint representations would be a step toward co-processing more diverse multimodal information. In this work, we present OmniBind, large-scale multimodal joint representation models ranging in scale from 7 billion to 30 billion parameters, which support 3D, audio, image, and language inputs. Due to the scarcity of data pairs across all modalities, instead of training large models from scratch, we propose remapping and binding the spaces of various pre-trained specialist models together. This approach enables "scaling up" by indirectly increasing the model parameters and the amount of seen data. To effectively integrate various spaces, we dynamically assign weights to different spaces by learning routers with two objectives: cross-modal overall alignment and language representation decoupling. Notably, since binding and routing spaces both only require lightweight networks, OmniBind is extremely training-efficient. Learning the largest 30B model requires merely unpaired unimodal data and approximately 3 days on a single 8-4090 node. Extensive experiments demonstrate the versatility and superiority of OmniBind as an omni representation model, highlighting its great potential for diverse applications, such as any-query and composable multimodal understanding.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。