Diffusion models (DMs) have become the new trend of generative models and have demonstrated a powerful ability of conditional synthesis. Among those, text-to-image diffusion models pre-trained on large-scale image-text pairs are highly controllable by customizable prompts. Unlike the unconditional generative models that focus on low-level attributes and details, text-to-image diffusion models contain more high-level knowledge thanks to the vision-language pre-training. In this paper, we propose VPD (Visual Perception with a pre-trained Diffusion model), a new framework that exploits the semantic information of a pre-trained text-to-image diffusion model in visual perception tasks. Instead of using the pre-trained denoising autoencoder in a diffusion-based pipeline, we simply use it as a backbone and aim to study how to take full advantage of the learned knowledge. Specifically, we prompt the denoising decoder with proper textual inputs and refine the text features with an adapter, leading to a better alignment to the pre-trained stage and making the visual contents interact with the text prompts. We also propose to utilize the cross-attention maps between the visual features and the text features to provide explicit guidance. Compared with other pre-training methods, we show that vision-language pre-trained diffusion models can be faster adapted to downstream visual perception tasks using the proposed VPD. Extensive experiments on semantic segmentation, referring image segmentation and depth estimation demonstrates the effectiveness of our method. Notably, VPD attains 0.254 RMSE on NYUv2 depth estimation and 73.3% oIoU on RefCOCO-val referring image segmentation, establishing new records on these two benchmarks. Code is available at https://github.com/wl-zhao/VPD
Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation
GenAI, Meta(Meta GenAI部门);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Sungkyunkwan University(成均馆大学);Georgia Institute of Technology(佐治亚理工学院)
WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation
National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学);University of Maryland, College Park(马里兰大学学院公园分校);Zhejiang University(浙江大学)
Unleashing the Capabilities of Large Vision-Language Models for Intelligent Perception of Roadside Infrastructure
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS)(信息工程测绘遥感国家重点实验室);Wuhan University(武汉大学);Hubei Luojia Laboratory(湖北珞珈实验室)
Unleashing Perception-Time Scaling to Multimodal Reasoning Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学北京校区人工智能学院);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大模型与智能治理重点实验室);ByteDance(字节跳动);University of California, San Diego(加州大学圣地亚哥分校);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models
Fudan University(复旦大学);Meituan(美团);Shanghai Key Lab of Intell. Info. Processing(上海智能信息处理重点实验室);Shanghai Collaborative Innovation Center on Intelligent Visual Computing(上海智能视觉计算协同创新中心)
Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models
Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算机系);Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative, Institute of Digital Twin, EIT(宁波空间智能与数字衍生关键实验室,数字孪生研究院,EIT);Shanghai Jiao Tong University(上海交通大学);Ocean University of China(中国海洋大学)
VisionKG: Unleashing the Power of Visual Datasets via Knowledge Graph
Technical University of Berlin(柏林工业大学);Bosch Center for Artificial Intelligence(博世人工智能中心);Fraunhofer Institute for Open Communication Systems(弗劳恩霍夫开放通信系统研究所)
Diff-Restorer: Unleashing Visual Prompts for Diffusion-based Universal Image Restoration
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University(上海交通大学图像通信与网络工程研究所);MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University(上海交通大学人工智能研究院、教育部人工智能重点实验室)
Visual Token Codec: Unleashing Spatial Redundancy for ViT Feature Coding
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);Pengcheng Laboratory(鹏城实验室);Ant Group(蚂蚁集团)
CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoning
School of Computer Science and Technology, East China Normal University(东华大学计算机科学与技术学院);King Abdullah University of Science and Technology(科廷大学);Fudan University(复旦大学)
Unleashing the Power of Vision-Language Models for Long-Tailed Multi-Label Visual Recognition
School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Key Laboratory of Computer Network and Information Integration (Southeast University)(计算机网络与信息集成重点实验室);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)(Mohamed bin Zayed人工智能大学);Carnegie Mellon University(卡内基梅隆大学)
Unleashing the Power of Pre-trained Encoders for Universal Adversarial Attack Detection
Faculty of Data Science, City University of Macau(澳门城市大学数据科学学院);Torrens University Australia(澳大利亚托伦斯大学);School of Computer Science and Engineering, Wuhan Institute of Technology(武汉工程大学计算机科学与工程学院)
GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognition
School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院);School of Computer Science, The University of Sydney(悉尼大学计算机科学学院);Department of Computer Science and Technology, Nanjing University(南京大学计算机科学与技术系);Artificial Intelligence Industrial Technology Research Institute, Nanjing Institute of Technology(南京工程学院人工智能产业技术研究院)
HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding
Tsinghua University(清华大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Johns Hopkins University(约翰斯·霍普金斯大学);SenseTime Research(商汤科技研究院);Nanjing University(南京大学)
MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigation
Fujian Key Laboratory of Urban Intelligent Sensing and Computing(福建智能感知与计算重点实验室);Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China(多媒体可信感知与高效计算重点实验室);Beihang University(北航);Nanyang Technological University(南洋理工大学);University of Chinese Academy of Sciences(中国科学院大学);Zhongguancun Academy(中关村学院)
EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild
The University of Hong Kong(香港大学);ShanghaiTech University(上海科技大学);Hong Kong University of Science and Technology(香港科学大学);Nanyang Technological University(南洋理工大学);Max Planck Institute for Informatics(马克斯·普朗克研究所(信息学));Texas A&M University(德克萨斯大学奥斯汀分校)
Unleashing Diffusion and State Space Models for Medical Image Segmentation
Department of Biostatistics, School of Global Public Health, New York University(生物统计学系、全球公共卫生学院、纽约大学);School of Statistics, KLATASDS-MOE, East China Normal University(统计学系、KLATASDS-MOE、华东师范大学);School of Biomedical Engineering, Southern Medical University(生物医学工程学院、南方医科大学);Faculty of Biomedical Engineering, Shenzhen University of Advanced Technology(生物医学工程学院、深圳先进技术大学)
Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials
Fudan University(复旦大学);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学);Stanford University(斯坦福大学);Nanyang Technological University(南洋理工大学)