We present a generative model of images that explicitly reasons over the set of objects they show. Our model learns a structured latent representation that separates objects from each other and from the background; unlike prior works, it explicitly represents the 2D position and depth of each object, as well as an embedding of its segmentation mask and appearance. The model can be trained from images alone in a purely unsupervised fashion without the need for object masks or depth information. Moreover, it always generates complete objects, even though a significant fraction of training images contain occlusions. Finally, we show that our model can infer decompositions of novel images into their constituent objects, including accurate prediction of depth ordering and segmentation of occluded parts.
University of Amsterdam（阿姆斯特丹大学）；The Netherlands Cancer Institute（荷兰癌症研究所）；Technical University of Denmark（丹麦技术大学）；University of Copenhagen（哥本哈根大学）；Copenhagen University Hospital（哥本哈根大学医院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Understanding which inductive biases could be helpful for the unsupervised learning of object-centric representations of natural scenes is challenging. In this paper, we systematically investigate the performance of two models on datasets where neural style transfer was used to obtain objects with complex textures while still retaining ground-truth annotations. We find that by using a single module to reconstruct both the shape and visual appearance of each object, the model learns more useful representations and achieves better object separation. In addition, we observe that adjusting the lat…
What 30,000 Hours of Ego-centric Video Does Not Teach
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Carnegie Mellon University（卡内基梅隆大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Toyota Research Institute（丰田研究所）
BEACON: Behavior and Appearance Control for Subject-Specific Video Generation
LIVIA, École de technologie supérieure（LIVIA，蒙特利尔高等技术学院）；CentraleSupélec, Université Paris-Saclay（巴黎中央理工-高等电力学院，巴黎-萨克雷大学）；Inria, Université Côte d’Azur（法国国家信息与自动化研究所，蔚蓝海岸大学）
DriveEditor: A Unified 3D Information-Guided Framework for Controllable Object Editing in Driving Scenes
Huazhong University of Science and Technology（华中科技大学）；National Key Laboratory of Multispectral Information Intelligent Processing Technology（多光谱信息智能处理技术国家重点实验室）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）
Polytechnique Montréal（蒙特利尔高等理工学院）；Institute of Biomedical Engineering, Polytechnique Montréal（蒙特利尔高等理工学院生物医学工程研究所）
We propose a novel unsupervised approach based on a two-stage object-centric adversarial framework that only needs object regions for detecting frame-level local anomalies in videos. The first stage consists in learning the correspondence between the current appearance and past gradient images of objects in scenes deemed normal, allowing us to either generate the past gradient from current appearance or the reverse. The second stage extracts the partial reconstruction errors between real and generated images (appearance and past gradient) with normal object behaviour, and trains a discriminato…
Learning Interactive World Model for Object-Centric Reinforcement Learning
University of California San Diego（加州大学圣地亚哥分校）；Mohamed bin Zayed University of Artificial Intelligence（马尔代夫穆罕默德·本·扎耶德人工智能大学）；University of Amsterdam（阿姆斯特丹大学）
Learning object-centric representations of multi-object scenes is a promising approach towards machine intelligence, facilitating high-level reasoning and control from visual sensory data. However, current approaches for unsupervised object-centric scene representation are incapable of aggregating information from multiple observations of a scene. As a result, these "single-view" methods form their representations of a 3D scene based only on a single 2D observation (view). Naturally, this leads to several inaccuracies, with these methods falling victim to single-view spatial ambiguities. To ad…
The visual world can be parsimoniously characterized in terms of distinct entities with sparse interactions. Discovering this compositional structure in dynamic visual scenes has proven challenging for end-to-end computer vision approaches unless explicit instance-level supervision is provided. Slot-based models leveraging motion cues have recently shown great promise in learning to represent, segment, and track objects without direct supervision, but they still fail to scale to complex real-world multi-object videos. In an effort to bridge this gap, we take inspiration from human development …
Adobe Research（Adobe研究院）；University of Surrey（萨里大学）
Images tell powerful stories but cannot always be trusted. Matching images back to trusted sources (attribution) enables users to make a more informed judgment of the images they encounter online. We propose a robust image hashing algorithm to perform such matching. Our hash is sensitive to manipulation of subtle, salient visual details that can substantially change the story told by an image. Yet the hash is invariant to benign transformations (changes in quality, codecs, sizes, shapes, etc.) experienced by images during online redistribution. Our key contribution is OSCAR-Net (Object-centric…
Inception Institute of Artificial Intelligence（起源人工智能研究院）；University of Bucharest（布加勒斯特大学）；SecurifAI
Abnormal event detection in video is a challenging vision problem. Most existing approaches formulate abnormal event detection as an outlier detection task, due to the scarcity of anomalous data during training. Because of the lack of prior information regarding abnormal events, these methods are not fully-equipped to differentiate between normal and abnormal events. In this work, we formalize abnormal event detection as a one-versus-rest binary classification problem. Our contribution is two-fold. First, we introduce an unsupervised feature learning framework based on object-centric convoluti…
Georgia Institute of Technology（佐治亚理工学院）；Toyota Research Institute（丰田研究院）
Our method studies the complex task of object-centric 3D understanding from a single RGB-D observation. As it is an ill-posed problem, existing methods suffer from low performance for both 3D shape and 6D pose and size estimation in complex multi-object scenarios with occlusions. We present ShAPO, a method for joint multi-object detection, 3D textured reconstruction, 6D object pose and size estimation. Key to ShAPO is a single-shot pipeline to regress shape, appearance and pose latent codes along with the masks of each object instance, which is then further refined in a sparse-to-dense fashion…
TrajGANR: Trajectory-Centric Urban Multimodal Learning via Geospatially Aligned Neural Representations
Google Research, Mountain View, CA（谷歌研究，山景城，加利福尼亚州）；Google LLC, Mountain View, CA（谷歌公司，山景城，加利福尼亚州）；Dept. of Computer Science, University of Southern California, Los Angeles, CA（计算机科学系，南加州大学，洛杉矶，加利福尼亚州）；SEAI Lab, Dept. of Geography and the Environment, The University of Texas at Austin, Austin, TX（SEAI实验室，地理与环境系，德克萨斯大学奥斯汀分校，奥斯汀，德克萨斯州）
