Beyond high-fidelity image synthesis, diffusion models have recently exhibited promising results in dense visual perception tasks. However, most existing work treats diffusion models as a standalone component for perception tasks, employing them either solely for off-the-shelf data augmentation or as mere feature extractors. In contrast to these isolated and thus sub-optimal efforts, we introduce a unified, versatile, diffusion-based framework, Diff-2-in-1, that can simultaneously handle both multi-modal data generation and dense visual perception, through a unique exploitation of the diffusion-denoising process. Within this framework, we further enhance discriminative visual perception via multi-modal generation, by utilizing the denoising network to create multi-modal data that mirror the distribution of the original training set. Importantly, Diff-2-in-1 optimizes the utilization of the created diverse and faithful data by leveraging a novel self-improving learning mechanism. Comprehensive experimental evaluations validate the effectiveness of our framework, showcasing consistent performance improvements across various discriminative backbones and high-quality multi-modal data generation characterized by both realism and usefulness. Our project website is available at https://zsh2000.github.io/diff-2-in-1.github.io/.
SpatialReward: Bridging the Perception Gap in Online RL for Image Editing via Explicit Spatial Reasoning
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));The Hong Kong University of Science and Technology(香港科学与技术大学);Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生学院,清华大学)
Sparse 3D Perception for Rose Harvesting Robots: A Two-Stage Approach Bridging Simulation and Real-World Applications
School of Computer Engineering, Iran University of Science and Technology(伊朗科学技术大学计算机工程学院);George Mason University Department of Computer Science(乔治·梅奥大学计算机科学系)
VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMs
Om AI Research(Om AI研究机构);Binjiang Institute of Zhejiang University(浙江大学滨江研究院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)
KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering
University of Science and Technology of China(中国科学技术大学);Data Darkness Lab, MIRACLE Center, USTC(数据黑暗实验室,MIRACLE中心,中国科学技术大学);School of Computer Science and Technology, Xidian University(西安电子科技大学计算机科学与技术学院)
UniV2D: Bridging Visual Restoration and Semantic Perception for Underwater Salient Object Detection
School of Computer Science, Wuhan University(武汉大学计算机学院);School of Computer Science, The University of Sydney(悉尼大学计算机学院);School of Computer Science and Technology, Harbin Institute of Technology(哈尔滨工业大学计算机科学与技术学院)
Unveiling and Bridging the Functional Perception Gap in MLLMs: Atomic Visual Alignment and Hierarchical Evaluation via PET-Bench
School of Biomedical Engineering, Southern Medical University(生物医学工程学院,南方医科大学);School of Biomedical Engineering, Shanghai Jiaotong University(生物医学工程学院,上海交通大学);Department of Electronic Engineering, Chinese University of Hong Kong(电子工程系,中国香港大学);Faculty of Dentistry, The University of Hong Kong(牙科学院,香港大学);Department of Nuclear Medicine, The Second Affiliated Hospital of Guangzhou University of Chinese Medicine(核医学科,广州中医药大学第二附属医院);PET Center, Department of Nuclear Medicine, Guangdong Provincial People’s Hospital, Southern Medical University(PET中心,核医学科,广东省人民医院,南方医科大学);Department of Nuclear Medicine, Nanfang Hospital, Southern Medical University(核医学科,南芳医院,南方医科大学);Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospitals(核医学与分子影像学部,日内瓦大学医院);Departments of Radiology, Physics, and Biomedical Engineering, The University of British Columbia(放射学、物理和生物医学工程系,不列颠哥伦比亚大学);Medical Artificial Intelligence Laboratory, Westlake University(医学人工智能实验室,西湖大学)
Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehension
Institute of Deep Perception Technology, JITRI(JITRI深度感知技术研究所);University of Liverpool(利物浦大学);University of Southampton(南安普顿大学);Momoni AI(Momoni人工智能公司);Xi’an Jiaotong-Liverpool University(西交利物浦大学);Thrust of Artificial Intelligence, HKUST (GZ)(香港科技大学(广州)人工智能学域);Thrust of Intelligent Transportation, HKUST (GZ)(香港科技大学(广州)智能交通学域)
Dance Across Shifts: Forward-Facilitation Continual Test-Time Adaptation through Dynamic Style Bridging
Harbin Institute of Technology(哈尔滨工业大学);Pengcheng Laboratory(鹏城实验室);Shenzhen University of Advanced Technology(深圳先进技术大学);Guangdong Provincial Key Laboratory of Computility Microelectronics(广东省计算微电子重点实验室)
Bridging Visual Affective Gap: Borrowing Textual Knowledge by Learning from Noisy Image-Text Pairs
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);TMCC, College of Computer Science, Nankai University(TMCC,南开大学计算机学院)
CM-Bench: A Comprehensive Cross-Modal Feature Matching Benchmark Bridging Visible and Infrared Images
School of Instrument Science and Opto-Electronics Engineering, Beijing Information Science and Technology University(北京信息科技大学仪器科学与光电工程学院);Department of Precision Instrument, Tsinghua University(清华大学精密仪器系)