Implicit and Explicit Language Guidance for Diffusion-based Visual Perception
School of Electrical and Information Engineering, Tianjin University(天津大学电气与信息工程学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);School of Big Data and Software Engineering, Chongqing University(重庆大学大数据与软件学院)
Hefeng Wang, Jiale Cao, Jin Xie, Aiping Yang, Yanwei Pang
Text-to-image diffusion models have shown powerful ability on conditional image synthesis. With large-scale vision-language pre-training, diffusion models are able to generate high-quality images with rich texture and reasonable structure under different text prompts. However, it is an open problem to adapt the pre-trained diffusion model for visual perception. In this paper, we propose an implicit and explicit language guidance framework for diffusion-based perception, named IEDP. Our IEDP comprises an implicit language guidance branch and an explicit language guidance branch. The implicit branch employs frozen CLIP image encoder to directly generate implicit text embeddings that are fed to diffusion model, without using explicit text prompts. The explicit branch utilizes the ground-truth labels of corresponding images as text prompts to condition feature extraction of diffusion model. During training, we jointly train diffusion model by sharing the model weights of these two branches. As a result, implicit and explicit branches can jointly guide feature learning. During inference, we only employ implicit branch for final prediction, which does not require any ground-truth labels. Experiments are performed on two typical perception tasks, including semantic segmentation and depth estimation. Our IEDP achieves promising performance on both tasks. For semantic segmentation, our IEDP has the mIoU$^\text{ss}$ score of 55.9% on AD20K validation set, which outperforms the baseline method VPD by 2.2%. For depth estimation, our IEDP outperforms the baseline method VPD with a relative gain of 11.0%.
Rethinking Driving World Model as Synthetic Data Generator for Perception Tasks
Peking University(北京大学);Xiaomi EV(小米电动车);Huazhong University of Science and Technology(华中科技大学);Beijing Key Laboratory of Data Intelligence and Security (Peking University)(北京数据智能与安全重点实验室(北京大学));Zhongguancun Academy(中关村学院)
NDM: A Noise-driven Detection and Mitigation Framework against Implicit Sexual Intentions in Text-to-Image Generation
Institute of Artificial Intelligence, Beihang University(北航人工智能研究院);College of AI, Tsinghua University(清华人工智能学院);Security Group, Alibaba Group(阿里集团安全组)
Implicit Guidance and Explicit Representation of Semantic Information in Points Cloud: A Survey
School of Information and Communication Engineering, Beijing University of Posts and Telecommunications (BUPT)(北京邮电大学信息与通信工程学院);Key Laboratory of Trustworthy Distributed Computing and Service (BUPT), Ministry of Education, China(教育部可信分布式计算与服务重点实验室)
DifferSeg: Towards Diverse Multimodal Binary Segmentation via Differential Perception and Frequency Guidance
School of Artificial Intelligence, Jiangxi Normal University(江西师范大学人工智能学院);Institute of AI Education, East China Normal University(华东师范大学人工智能教育研究所);Yale School of Medicine, Yale University(耶鲁大学医学院)
Crash Report Data Analysis for Creating Scenario-Wise, Spatio-Temporal Attention Guidance to Support Computer Vision-based Perception of Fatal Crash Risks
AREA3D: Active Reconstruction Agent with Unified Feed-Forward 3D Perception and Vision-Language Guidance
Southern University of Science and Technology(南方科技大学);Harvard University(哈佛大学);California Institute of Technology(加州理工学院);Massachusetts Institute of Technology(麻省理工学院)
Unveiling the Potential of Segment Anything Model 2 for RGB-Thermal Semantic Segmentation with Language Guidance
School of Artificial Intelligence and Robotics, Hunan University(人工智能与机器人学院,湖南大学);National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University(机器人视觉感知与控制技术国家工程研究中心,湖南大学);AI Thrust, Hong Kong University of Science and Technology (Guangzhou)(香港理工大学(广州)人工智能方向)
VOIC: Visible-Occluded Integrated Guidance for 3D Semantic Scene Completion
Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology(可信自主系统研究院,南方科技大学);Department of Computer Science and Engineering, Southern University of Science and Technology(计算机科学与工程系,南方科技大学);School of Computer Science, University of Nottingham Ningbo China(宁波大学计算机学院);Department of Electronic and Information Engineering, Changchun University(电子与信息工程学院,长春大学)
Exploiting Completeness Perception with Diffusion Transformer for Unified 3D MRI Synthesis
School of Engineering, University of Birmingham, UK(伯明翰大学工程学院);William Harvey Research Institute, Queen Mary University London, UK(女王玛丽大学伦敦威廉·哈里维研究所);Barts Heart Centre, St Bartholomew’s Hospital, Barts Health NHS Trust, UK(巴特勒心脏中心,圣巴塞洛缪医院,巴特勒健康 NHS信托);Division of Cardiology, Johns Hopkins University School of Medicine, US(约翰霍普金斯大学医学院心脏病科)