多模态模型正变得越来越有效,部分原因在于诸如 Transformer 架构等统一组件。然而,多模态模型通常仍由许多特定于任务和模态的组件及训练流程组成。例如,CLIP (Radford et al., 2021) 通过对比损失训练独立的文本和图像塔。我们探索了一种额外的统一方式:使用纯基于像素的模型来执行图像、文本和多模态任务。我们的模型仅使用对比损失进行训练,因此我们将其称为 CLIP-Pixels Only (CLIPPO)。CLIPPO 使用单一编码器处理常规图像和渲染为图像的文本。CLIPPO 在检索和零样本图像分类等基于图像的任务上表现几乎与 CLIP 风格模型一样好,且参数量仅为后者的一半,没有特定于文本的塔或嵌入。当通过图像-文本对比学习和下一句对比学习进行联合训练时,CLIPPO 能在自然语言理解任务上表现良好,且无需任何词级别的损失(语言建模或掩码语言建模),优于基于像素的先前工作。令人惊讶的是,CLIPPO 仅通过将问题和图像一起渲染,就能在视觉问答中获得良好的准确率。最后,我们利用 CLIPPO 不需要分词器这一事实,表明它无需修改即可在多语言多模态检索上取得强劲表现。
英文摘要:
Multimodal models are becoming increasingly effective, in part due to unified components, such as the Transformer architecture. However, multimodal models still often consist of many task- and modality-specific pieces and training procedures. For example, CLIP (Radford et al., 2021) trains independent text and image towers via a contrastive loss. We explore an additional unification: the use of a pure pixel-based model to perform image, text, and multimodal tasks. Our model is trained with contrastive loss alone, so we call it CLIP-Pixels Only (CLIPPO). CLIPPO uses a single encoder that processes both regular images and text rendered as images. CLIPPO performs image-based tasks such as retrieval and zero-shot image classification almost as well as CLIP-style models, with half the number of parameters and no text-specific tower or embedding. When trained jointly via image-text contrastive learning and next-sentence contrastive learning, CLIPPO can perform well on natural language understanding tasks, without any word-level loss (language modelling or masked language modelling), outperforming pixel-based prior work. Surprisingly, CLIPPO can obtain good accuracy in visual question answering, simply by rendering the question and image together. Finally, we exploit the fact that CLIPPO does not require a tokenizer to show that it can achieve strong performance on multilingual multimodal retrieval without modifications.
Image Bitstream Fine-grained Understanding for Privacy-Friendly AIoT
Huazhong University of Science and Technology(华中科技大学);Nanyang Technological University(南洋理工大学);Central South University(中南大学);Harbin Engineering University(哈尔滨工程大学)
UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning
The Hong Kong Polytechnic University(香港理工大学);ARC Lab, Tencent PCG(腾讯PCG ARC实验室);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);vivo Mobile Communication Co.(vivo移动通信公司);MindWingman Technology (Shenzhen) Co., Ltd.(深圳MindWingman技术有限公司)
Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level
University of Central Florida(中佛罗里达大学);University of Western Australia(西澳大学);Amazon Web Services(亚马逊网络服务);The University of Texas at Dallas(德克萨斯大学达拉斯分校)
IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation
Zhejiang University(浙江大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Ningbo Global Innovation Center, Zhejiang University(浙江大学宁波全球创新中心);Zhejiang Key Laboratory of Digital-Intelligence Service Technology(浙江省数字智能服务技术重点实验室)
Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs
New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA)(模式识别新实验室、多模态人工智能系统国家重点实验室、自动化研究所、中国科学院(CASIA));University of Chinese Academy of Sciences(中国科学院大学);Peking University(北京大学);Wuhan University(武汉大学);ByteDance(字节跳动)
On the Design Fundamentals of Pixel Text Representation Learning
The Chinese University of Hong Kong(香港中文大学);DAMO Academy, Alibaba Group(阿里巴巴达摩院);Fudan University(复旦大学);Aberystwyth University(阿伯里斯特威斯大学);University of Macau(澳门大学);Shanghai University of Finance and Economics(上海财经大学)
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs
Allen Institute for Artificial Intelligence(艾伦人工智能研究所);Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院)
Beyond Pixels: A Vector-to-Graph Framework for Reliable Schematic Auditing
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(人工智能与数字经济广东实验室);Guangdong Power Grid Co., Ltd.(广东电网公司);Shanghai University(上海大学);Carleton University(卡尔顿大学)
SurgMLLMBench: A Multimodal Large Language Model Benchmark Dataset for Surgical Scene Understanding
Samsung Research(三星研究所);Center for Humanoid Research, Korea Institute of Science and Technology(人类研究学院,韩国科学技术院);Department of plastic surgery, College of medicine, Korea University(医学院整形外科部,韩国大学);Department of orthopedic surgery, College of medicine, Korea University(医学院骨科部,韩国大学)
UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Beihang University(北京航空航天大学);Shenzhen Key Lab of Computer Vision and Pattern Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(深圳计算机视觉与模式识别重点实验室,深圳先进技术研究院,中国科学院);Nanjing University(南京大学);University of Science and Technology of China(中国科学技术大学);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
1st Place Winner of the 2024 Pixel-level Video Understanding in the Wild (CVPR'24 PVUW) Challenge in Video Panoptic Segmentation and Best Long Video Consistency of Video Semantic Segmentation
Device Solutions Research America, Samsung Semiconductor, Inc.(三星半导体公司美国设备解决方案研究中心)
PixelVLA: Advancing Pixel-level Understanding in Vision-Language-Action Model
University of Trento(特伦托大学);School of Automation Science and Engineering, South China University of Technology(华南理工大学自动化科学与工程学院);Mohamed bin Zayed University of Artificial Intelligence(马尔代夫人工智能大学);Australian National University(澳大利亚国立大学)
Every Pixel Counts: Unsupervised Geometry Learning with Holistic 3D Motion Understanding
University of Southern California(南加州大学);Baidu Research(百度研究);National Engineering Laboratory for Deep Learning Technology and Applications(深度学习技术及应用国家工程实验室)