In the battle against widespread online misinformation, a growing problem is text-image inconsistency, where images are misleadingly paired with texts with different intent or meaning. Existing classification-based methods for text-image inconsistency can identify contextual inconsistencies but fail to provide explainable justifications for their decisions that humans can understand. Although more nuanced, human evaluation is impractical at scale and susceptible to errors. To address these limitations, this study introduces D-TIIL (Diffusion-based Text-Image Inconsistency Localization), which employs text-to-image diffusion models to localize semantic inconsistencies in text and image pairs. These models, trained on large-scale datasets act as ``omniscient" agents that filter out irrelevant information and incorporate background knowledge to identify inconsistencies. In addition, D-TIIL uses text embeddings and modified image regions to visualize these inconsistencies. To evaluate D-TIIL's efficacy, we introduce a new TIIL dataset containing 14K consistent and inconsistent text-image pairs. Unlike existing datasets, TIIL enables assessment at the level of individual words and image regions and is carefully designed to represent various inconsistencies. D-TIIL offers a scalable and evidence-based approach to identifying and localizing text-image inconsistency, providing a robust framework for future research combating misinformation.
Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Rendering
University of California, Irvine(加州大学伊维特分校);Stony Brook University(石溪大学);Huazhong University of Science and Technology(华中科技大学);University of Florida(佛罗里达大学)
Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidance
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);SenseTime Research(商汤科技研究院);The University of Hong Kong(香港大学);The Chinese University of Hong Kong(香港中文大学)
Exploring Specular Reflection Inconsistency for Generalizable Face Forgery Detection
School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室);WeChat AI, Tencent Inc(腾讯公司微信AI)
Inconsistency-aware Multimodal Schrödinger Bridge for Deepfake Localization
Department of Computer Science and Techonology, Huaqiao University(华侨大学计算机科学与技术系);Xiamen Key Laboratory of Computer Vision and Pattern Recognition, Huaqiao University(厦门计算机视觉与模式识别重点实验室);Tongji University(同济大学);School of Cyber Science and Engineering, Wuhan University(武汉大学网络空间安全学院)
MMA-Diffusion: MultiModal Attack on Diffusion Models
The Chinese University of Hong Kong(香港中文大学);Huawei(华为);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Beijing Wenge Technology Co. Ltd(北京文格科技有限公司)
DiGS-Avatar: Single-Image Animatable 3D Human Reconstruction via UV-Space Diffusion
Key Laboratory of Media Audio and Video (Communication University of China)(中国传媒大学媒体音频与视频重点实验室);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院)
Diffusion Model is a Good Pose Estimator from 3D RF-Vision
School of Electrical and Electronic Engineering(电气与电子工程学院);School of Mechanical and Aerospace Engineering(机械与航空航天工程学院);Nanyang Technological University(南洋理工大学)
Towards Robust Defense against Customization via Protective Perturbation Resistant to Diffusion-based Purification
MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences(自动化研究所、中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院、中国科学院大学)
MTADiffusion: Mask Text Alignment Diffusion Model for Object Inpainting
MT Lab, Meitu Inc(美图实验室,美图公司);National University of Singapore(新加坡国立大学);Department of Computer Science and Technology, BNRist, IDG/McGovern Institute for Brain Research, Tsinghua University(计算机科学与技术系,BNRist,IDG/McGovern脑科学研究院,清华大学)
Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D Brain MRI Synthesis
The University of Hong Kong(香港大学);The Chinese University of Hong Kong(香港中文大学);Qilu Hospital of Shandong University(山东大学齐鲁医院);Nanyang Technological University(南洋理工大学)
LatentColorization: Latent Diffusion-Based Speaker Video Colorization
SFI Centre for Research Training in Artificial Intelligence(SFI人工智能研究培训中心);University of Galway(戈尔韦大学);School of Computer Science(计算机科学学院);School of Engineering(工程学院)
AttDiff-GAN: A Hybrid Diffusion-GAN Framework for Facial Attribute Editing
GuangDong Province Key Lab of Information Security Technology and School of Computer Science and Engineering(广东信息安全技术重点实验室和计算机科学与工程学院);School of Cyber Science and Technology, Shenzhen Campus(深圳校区网络科学与技术学院);Guangdong Laboratory of Machine Perception and Intelligent Computing, Faculty of Engineering(广东机器感知与智能计算实验室,工程学院)
Uniform Discrete Diffusion with Metric Path for Video Generation
National Laboratory of Pattern Recognition, CASIA(中国科学院自动化所模式识别国家实验室);Key Laboratory of Intelligent Information Processing, ICT, CAS(中国科学院信息科技重点实验室);University of Chinese Academy of Sciences(中国科学院大学);Zhejiang University(浙江大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院)