arXiv 2305.111722023-05-18cs.CVcs.CLcs.SDeess.AS

ONE-PEACE：探索面向无限模态的通用表示模型

ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities

DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Huazhong University of Science and Technology（华中科技大学）

Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, Chang Zhou

备注：: 30 pages, 9 figures, 18 tables

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出ONE-PEACE，一个具有4B参数的可扩展通用表示模型，通过模态适配器、共享自注意力层和模态FFN架构，以及两个模态无关的预训练任务，实现对视觉、音频和语言模态的无缝对齐与融合，并在众多单模态和多模态任务上取得领先结果。

摘要：

在这项工作中，我们探索了一种构建面向无限模态的通用表示模型的可扩展方法。我们发布了ONE-PEACE，这是一个具有4B参数的高度可扩展模型，能够无缝对齐和整合视觉、音频和语言模态的表示。ONE-PEACE的架构包括模态适配器、共享自注意力层和模态FFN。这种设计允许通过添加适配器和FFN轻松扩展新模态，同时也能通过自注意力层实现多模态融合。为了预训练ONE-PEACE，我们开发了两个与模态无关的预训练任务：跨模态对齐对比和模态内去噪对比，它们同时对齐不同模态的语义空间并捕获模态内的细粒度细节。凭借这种对扩展友好的架构和预训练任务，ONE-PEACE有潜力扩展到无限模态。在不使用任何视觉或语言预训练模型进行初始化的情况下，ONE-PEACE在广泛的单模态和多模态任务上取得了领先结果，包括图像分类（ImageNet）、语义分割（ADE20K）、音频文本检索（AudioCaps, Clotho）、音频分类（ESC-50, FSD50K, VGGSound）、音频问答（AVQA）、图像文本检索（MSCOCO, Flickr30K）和视觉定位（RefCOCO/+/g）。代码可在https://github.com/OFA-Sys/ONE-PEACE获取。

英文摘要：

In this work, we explore a scalable way for building a general representation model toward unlimited modalities. We release ONE-PEACE, a highly extensible model with 4B parameters that can seamlessly align and integrate representations across vision, audio, and language modalities. The architecture of ONE-PEACE comprises modality adapters, shared self-attention layers, and modality FFNs. This design allows for the easy extension of new modalities by adding adapters and FFNs, while also enabling multi-modal fusion through self-attention layers. To pretrain ONE-PEACE, we develop two modality-agnostic pretraining tasks, cross-modal aligning contrast and intra-modal denoising contrast, which align the semantic space of different modalities and capture fine-grained details within modalities concurrently. With the scaling-friendly architecture and pretraining tasks, ONE-PEACE has the potential to expand to unlimited modalities. Without using any vision or language pretrained model for initialization, ONE-PEACE achieves leading results on a wide range of uni-modal and multi-modal tasks, including image classification (ImageNet), semantic segmentation (ADE20K), audio-text retrieval (AudioCaps, Clotho), audio classification (ESC-50, FSD50K, VGGSound), audio question answering (AVQA), image-text retrieval (MSCOCO, Flickr30K), and visual grounding (RefCOCO/+/g). Code is available at https://github.com/OFA-Sys/ONE-PEACE.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。