The issue of generative pretraining for vision models has persisted as a long-standing conundrum. At present, the text-to-image (T2I) diffusion model demonstrates remarkable proficiency in generating high-definition images matching textual inputs, a feat made possible through its pre-training on large-scale image-text pairs. This leads to a natural inquiry: can diffusion models be utilized to tackle visual perception tasks? In this paper, we propose a simple yet effective scheme to harness a diffusion model for visual perception tasks. Our key insight is to introduce learnable embeddings (meta prompts) to the pre-trained diffusion models to extract proper features for perception. The effect of meta prompts are two-fold. First, as a direct replacement of the text embeddings in the T2I models, it can activate task-relevant features during feature extraction. Second, it will be used to re-arrange the extracted features to ensures that the model focuses on the most pertinent features for the task on hand. Additionally, we design a recurrent refinement training strategy that fully leverages the property of diffusion models, thereby yielding stronger visual features. Extensive experiments across various benchmarks validate the effectiveness of our approach. Our approach achieves new performance records in depth estimation tasks on NYU depth V2 and KITTI, and in semantic segmentation task on CityScapes. Concurrently, the proposed method attains results comparable to the current state-of-the-art in semantic segmentation on ADE20K and pose estimation on COCO datasets, further exemplifying its robustness and versatility.
VMBench: A Benchmark for Perception-Aligned Video Motion Generation
AMAP, Alibaba Group(阿里巴巴集团AMAP);CRISE, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所CRISE);School of Computer Science and Technology, University of Chinese Academy of Sciences(中国科学院大学计算机科学与技术学院)
Representation Learning in Diffusion and Flow-based Model: An Application Aspect
Fudan University(复旦大学);School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University(西北工业大学光学与电子信息人工智能学院(iOPEN));The University of Hong Kong(香港大学)
Reversible Unfolding Network for Concealed Visual Perception with Generative Refinement
Department of Biomedical Engineering, Duke University(生物医学工程系,杜克大学);Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Nankai International Advanced Research Institute (SHENZHEN FUTIAN), Nankai University(南开国际先进研究院(深圳福田),南开大学);College of Computer Science, Nankai University(计算机科学学院,南开大学)
Harnessing Meta-Learning for Controllable Full-Frame Video Stabilization
School of Computing and Artificial Intelligence(计算与人工智能学院);Southwest Jiaotong University(西南交通大学);Department of Computer Science(计算机科学系);Hanynag University(汉阳大学);School of Computing and Augmented Intelligence(计算与增强智能学院);Arizona State University(亚利桑那州立大学)
RS3DBench: A Comprehensive Benchmark for 3D Spatial Perception in Remote Sensing
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Software College of Zhejiang University(浙江大学软件学院);Hangzhou City University(杭州市大学)
Underwater Variable Zoom: Depth-Guided Perception Network for Underwater Image Enhancement
School of Computer Science and Technology, Dalian University of Technology(大连理工大学计算机科学与技术学院);University of New South Wales(新南威尔士大学);School of Computer Science and Technology, Shandong Technology and Business University(山东工商学院计算机科学与技术学院);School of Information and Communication Engineering, Dalian Minzu University(大连民族大学信息与通信工程学院);School of Innovation Experiment, Dalian University of Technology(大连理工大学创新实验学院)
NeuroVoxel-LM: Language-Aligned 3D Perception via Dynamic Voxelization and Meta-Embedding
School of Electrical and Electronic Engineering(电气与电子工程学院);Nanyang Technological University(南洋理工大学);School of Computer Science and Technology(计算机科学与技术学院);University of Chinese Academy of Sciences(中国科学院大学)