Most of computer vision focuses on what is in an image. We propose to train a standalone object-centric context representation to perform the opposite task: seeing what is not there. Given an image, our context model can predict where objects should exist, even when no object instances are present. Combined with object detection results, we can perform a novel vision task: finding where objects are missing in an image. Our model is based on a convolutional neural network structure. With a specially designed training strategy, the model learns to ignore objects and focus on context only. It is fully convolutional thus highly efficient. Experiments show the effectiveness of the proposed approach in one important accessibility task: finding city street regions where curb ramps are missing, which could help millions of people with mobility disabilities.
During 3D reconstruction, it is often the case that people cannot scan each individual object from all views, resulting in missing geometry in the captured scan. This missing geometry can be fundamentally limiting for many applications, e.g., a robot needs to know the unseen geometry to perform a precise grasp on an object. Thus, we introduce the task of semantic instance completion: from an incomplete RGB-D scan of a scene, we aim to detect the individual object instances and infer their complete object geometry. This will open up new possibilities for interactions with objects in a scene, fo…
Carnegie Mellon University（卡内基梅隆大学）；Facebook AI Research（Facebook人工智能研究院）；Microsoft Research（微软研究院）
When human annotators are given a choice about what to label in an image, they apply their own subjective judgments on what to ignore and what to mention. We refer to these noisy "human-centric" annotations as exhibiting human reporting bias. Examples of such annotations include image tags and keywords found on photo sharing sites, or in datasets containing image captions. In this paper, we use these noisy annotations for learning visually correct image classifiers. Such annotations do not use consistent vocabulary, and miss a significant amount of the information present in an image; however,…
Seeing Symbols, Missing Cultures: Probing Vision-Language Models' Reasoning on Fire Imagery and Cultural Meaning
Duncan of Jordanstone College of Art & Design (DJCAD), University of Dundee（邓迪大学邓迪艺术与设计学院（DJCAD））；Guangzhou Institute of Science and Technology (GZIST)（广州科学技术研究院）；Faculty of Arts, Xiamen University（厦门大学艺术学院）；University of Birmingham（伯明翰大学）
Seeing What Matters: Visual Preference Policy Optimization for Visual Generation
Southeast University（东南大学）；Institute of Artificial Intelligence (TeleAI), China Telecom（人工智能研究院（TeleAI），中国电信）；University of Science and Technology of China（中国科学技术大学）
Stanford University（斯坦福大学）；Massachusetts Institute of Technology（麻省理工学院）；Purdue University（普渡大学）
We present a system that demonstrates how the compositional structure of events, in concert with the compositional structure of language, can interplay with the underlying focusing mechanisms in video action recognition, thereby providing a medium, not only for top-down and bottom-up integration, but also for multi-modal integration between vision and language. We show how the roles played by participants (nouns), their characteristics (adjectives), the actions performed (verbs), the manner of such actions (adverbs), and changing spatial relations between participants (prepositions) in the for…
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）
Despite the success of Generative Adversarial Networks (GANs), mode collapse remains a serious issue during GAN training. To date, little work has focused on understanding and quantifying which modes have been dropped by a model. In this work, we visualize mode collapse at both the distribution level and the instance level. First, we deploy a semantic segmentation network to compare the distribution of segmented objects in the generated images with the target distribution in the training set. Differences in statistics reveal object classes that are omitted by a GAN. Second, given the identifie…
Cross-modal alignment is essential for vision-language pre-training (VLP) models to learn the correct corresponding information across different modalities. For this purpose, inspired by the success of masked language modeling (MLM) tasks in the NLP pre-training area, numerous masked modeling tasks have been proposed for VLP to further promote cross-modal interactions. The core idea of previous masked modeling tasks is to focus on reconstructing the masked tokens based on visible context for learning local-to-local alignment. However, most of them pay little attention to the global semantic fe…
Despite their excellent performance, state-of-the-art computer vision models often fail when they encounter adversarial examples. Video perception models tend to be more fragile under attacks, because the adversary has more places to manipulate in high-dimensional data. In this paper, we find one reason for video models' vulnerability is that they fail to perceive the correct motion under adversarial perturbations. Inspired by the extensive evidence that motion is a key factor for the human visual system, we propose to correct what the model sees by restoring the perceived motion information. …
Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs
The University of Manchester（曼彻斯特大学）；The University of Melbourne（墨尔本大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Edinburgh（爱丁堡大学）；University of Southern California（南加州大学）；Fudan University（复旦大学）；University of Newcastle（纽卡斯尔大学）
University of Electronic Science and Technology of China（电子科技大学）；National University of Singapore（新加坡国立大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of Science and Technology Beijing（北京科技大学）
Talking face generation, also known as speech-to-lip generation, reconstructs facial motions concerning lips given coherent speech input. The previous studies revealed the importance of lip-speech synchronization and visual quality. Despite much progress, they hardly focus on the content of lip movements i.e., the visual intelligibility of the spoken words, which is an important aspect of generation quality. To address the problem, we propose using a lip-reading expert to improve the intelligibility of the generated lip regions by penalizing the incorrect generation results. Moreover, to compe…
Bimanual gestures are of the utmost importance for the study of motor coordination in humans and in everyday activities. A reliable detection of bimanual gestures in unconstrained environments is fundamental for their clinical study and to assess common activities of daily living. This paper investigates techniques for a reliable, unconstrained detection and classification of bimanual gestures. It assumes the availability of inertial data originating from the two hands/arms, builds upon a previously developed technique for gesture modelling based on Gaussian Mixture Modelling (GMM) and Gaussia…
TextVerifier: Seeing Beyond Recognition to Verify Chinese Text Rendering
State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能全国重点实验室）；Alibaba Group（阿里巴巴集团）；State Key Lab of CAD&CG, Zhejiang University（浙江大学计算机辅助设计与图形学国家重点实验室）
Convolution has been arguably the most important feature transform for modern neural networks, leading to the advance of deep learning. Recent emergence of Transformer networks, which replace convolution layers with self-attention blocks, has revealed the limitation of stationary convolution kernels and opened the door to the era of dynamic feature transforms. The existing dynamic transforms, including self-attention, however, are all limited for video understanding where correspondence relations in space and time, i.e., motion information, are crucial for effective representation. In this wor…
Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalities
IMT School for Advanced Studies of Lucca（卢卡高等研究院）；Institute of Computational Perception, Johannes Kepler University（约翰开普勒大学计算感知研究所）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Engineering and Technology, Taxila（塔克西拉工程技术大学）；Dalhousie University（戴尔豪斯大学）；Interuniversity Microelectronics Centre (IMEC)（微电子学研究中心）；Linz Institute of Technology（林茨理工学院）
Artificial intelligence (AI) systems power the world we live in. Deep neural networks (DNNs) are able to solve tasks in an ever-expanding landscape of scenarios, but our eagerness to apply these powerful models leads us to focus on their performance and deprioritises our ability to understand them. Current research in the field of explainable AI tries to bridge this gap by developing various perturbation or gradient-based explanation techniques. For images, these techniques fail to fully capture and convey the semantic information needed to elucidate why the model makes the predictions it does…
Scene models allow robots to reason about what is in the scene, what else should be in it, and what should not be in it. In this paper, we propose a hybrid Boltzmann Machine (BM) for scene modeling where relations between objects are integrated. To be able to do that, we extend BM to include tri-way edges between visible (object) nodes and make the network to share the relations across different objects. We evaluate our method against several baseline models (Deep Boltzmann Machines, and Restricted Boltzmann Machines) on a scene classification dataset, and show that it performs better in sever…
What You Have is What You Track: Adaptive and Robust Multimodal Tracking
TeleAI, China Telecom（TeleAI，中国电信）；Computer Vision Lab, CAIDAS & IFI, University of Wurzburg（计算机视觉实验室，CAIDAS与IFI，乌尔姆大学）；INSAIT, Sofia University（INSAIT，索菲亚大学）；ShanghaiTech University（上海科技大学）；University of Copenhagen（哥本哈根大学）；AI Institute, Shanghai Jiao Tong University（人工智能研究院，上海交通大学）
Seeing is Believing: Aligning Prompt Rewriting with Visual Anchors for Text-to-Image Generation
Peking University（北京大学）；Tencent（腾讯）；Dalian University of Technology（大连理工大学）；Nanyang Technological University（南洋理工大学）；University of Cambridge（剑桥大学）；Zhejiang University（浙江大学）
Scene Parsing is a crucial step to enable autonomous systems to understand and interact with their surroundings. Supervised deep learning methods have made great progress in solving scene parsing problems, however, come at the cost of laborious manual pixel-level annotation. To alleviate this effort synthetic data as well as weak supervision have both been investigated. Nonetheless, synthetically generated data still suffers from severe domain shift while weak labels are often imprecise. Moreover, most existing works for weakly supervised scene parsing are limited to salient foreground objects…
Multiplex immunofluorescence (MxIF) is an emerging imaging technique that produces the high sensitivity and specificity of single-cell mapping. With a tenet of 'seeing is believing', MxIF enables iterative staining and imaging extensive antibodies, which provides comprehensive biomarkers to segment and group different cells on a single tissue section. However, considerable depletion of the scarce tissue is inevitable from extensive rounds of staining and bleaching ('missing tissue'). Moreover, the immunofluorescence (IF) imaging can globally fail for particular rounds ('missing stain''). In th…
Seeing is not always believing: Benchmarking Human and Model Perception of AI-Generated Images
Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；The University of Sydney（悉尼大学）
What is Missing? Explaining Neurons Activated by Absent Concepts
Max Planck Institute for Informatics, SIC（马克斯·普朗克信息研究所，SIC）；Department of Computer Science, Technical University of Darmstadt（达姆施塔特技术大学计算机科学系）；Leibniz Institute for Resilience Research（莱比锡韧性研究所）；Institute for Quantitative and Computational Biosciences, Johannes Gutenberg University Mainz（美因茨雅各布·冯·特利尔大学定量与计算生物科学研究所）；University Medical Center Mainz（美因茨大学医学中心）
IBM Research（IBM研究院）；University of Michigan（密歇根大学）
In this paper we propose a novel method that provides contrastive explanations justifying the classification of an input by a black box classifier such as a deep neural network. Given an input we find what should be %necessarily and minimally and sufficiently present (viz. important object pixels in an image) to justify its classification and analogously what should be minimally and necessarily \emph{absent} (viz. certain background pixels). We argue that such explanations are natural for humans and are used commonly in domains such as health care and criminology. What is minimally but critica…
Can We Talk Models Into Seeing the World Differently?
Offenburg University（奥芬堡大学）；University of Mannheim（曼海姆大学）；Tübingen AI Center（蒂宾根人工智能中心）；Goethe University Frankfurt（法兰克福大学）；University of Siegen（锡根大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Google DeepMind（谷歌DeepMind）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）
