The ability to perceive and reason about individual objects and their interactions is a goal to be achieved for building intelligent artificial systems. State-of-the-art approaches use a feedforward encoder to extract object embeddings and a latent graph neural network to model the interaction between these object embeddings. However, the feedforward encoder can not extract {\it object-centric} representations, nor can it disentangle multiple objects with similar appearance. To solve these issues, we introduce {\it Slot Structured World Models} (SSWM), a class of world models that combines an {\it object-centric} encoder (based on Slot Attention) with a latent graph-based dynamics model. We evaluate our method in the Spriteworld benchmark with simple rules of physical interaction, where Slot Structured World Models consistently outperform baselines on a range of (multi-step) prediction tasks with action-conditional object interactions. All code to reproduce paper experiments is available from \url{https://github.com/JonathanCollu/Slot-Structured-World-Models}.
Although DALL-E has shown an impressive ability of composition-based systematic generalization in image generation, it requires the dataset of text-image pairs and the compositionality is provided by the text. In contrast, object-centric representation models like the Slot Attention model learn composable representations without the text prompt. However, unlike DALL-E its ability to systematically generalize for zero-shot generation is significantly limited. In this paper, we propose a simple but novel slot-based autoencoding architecture, called SLATE, for combining the best of both worlds: l…
HeSRN: Representation Learning On Heterogeneous Graphs via Slot-Aware Retentive Network
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Zayed University（扎耶德大学）；Higher Colleges of Technology（高等技术学院）；Monash University（莫纳什大学）；Hangzhou Dianzi University（杭州电子科技大学）；University of Southampton（南安普顿大学）
We introduce the first metric for evaluating disentanglement at individual hierarchy levels of a structured latent representation. Applied to object-centric generative models, this offers a systematic, unified approach to evaluating (i) object separation between latent slots (ii) disentanglement of object properties inside individual slots (iii) disentanglement of intrinsic and extrinsic object properties. We theoretically show that for structured representations, our framework gives stronger guarantees of selecting a good model than previous disentanglement metrics. Experimentally, we demonst…
MultiBind: A Benchmark for Attribute Misbinding in Multi-Subject Generation
New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别新实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；The University of Chicago（芝加哥大学）；ByteDance（字节跳动）
Unsupervised extraction of objects from low-level visual data is an important goal for further progress in machine learning. Existing approaches for representing objects without labels use structured generative models with static images. These methods focus a large amount of their capacity on reconstructing unimportant background pixels, missing low contrast or small objects. Conversely, we present a new method that avoids losses in pixel space and over-reliance on the limited signal a static image provides. Our approach takes advantage of objects' motion by learning a discriminative, time-con…
Advancing Real-World Parking Slot Detection with Large-Scale Dataset and Semi-Supervised Baseline
Institute of Information Science, Beijing Jiaotong University（信息科学学院，北京交通大学）；Visual Intelligence +X International Cooperation Joint Laboratory of MOE（教育部视觉智能+X国际合作联合实验室）
Future Slot Prediction for Unsupervised Object Discovery in Surgical Video
PCASO Laboratory, Department of Surgery, University of Pennsylvania（匹兹堡大学外科实验室、外科系、匹兹堡大学）；Department of Computer and Information Science, University of Pennsylvania（计算机与信息科学系、匹兹堡大学）
FORLA: Federated Object-centric Representation Learning with Slot Attention
PCASO Laboratory, Dept. of Surgery, University of Pennsylvania（宾夕法尼亚大学外科部PCASO实验室）；Dept. of Computer and Information Science, University of Pennsylvania（宾夕法尼亚大学计算机与信息科学系）
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Samsung AI Centre Toronto（三星多伦多人工智能中心）；Google Research（谷歌研究院）
Understanding dynamics from visual observations is a challenging problem that requires disentangling individual objects from the scene and learning their interactions. While recent object-centric models can successfully decompose a scene into objects, modeling their dynamics effectively still remains a challenge. We address this problem by introducing SlotFormer -- a Transformer-based autoregressive model operating on learned object-centric representations. Given a video clip, our approach reasons over object features to model spatio-temporal relationships and predicts accurate future object s…
The visual world can be parsimoniously characterized in terms of distinct entities with sparse interactions. Discovering this compositional structure in dynamic visual scenes has proven challenging for end-to-end computer vision approaches unless explicit instance-level supervision is provided. Slot-based models leveraging motion cues have recently shown great promise in learning to represent, segment, and track objects without direct supervision, but they still fail to scale to complex real-world multi-object videos. In an effort to bridge this gap, we take inspiration from human development …
Slot-BERT: Self-supervised Object Discovery in Surgical Video
Outcomes Laboratory, Department of Surgery, University of Pennsylvania, Philadelphia, PA, USA.；Department of Computer；Information Science, University of Pennsylvania, Philadelphia, PA, USA.；Division of Thoracic Surgery, Toronto General Hospital, University Health Network, Toronto, Ontario, Canada.；Surgical Artificial Intelligence Research Academy, University Health Network, Toronto, ON, Canada.
Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；University of Amsterdam（阿姆斯特丹大学）；University of Tübingen（蒂宾根大学）；Archimedes/Athena RC（阿基米德/雅典娜研究中心）
SlotVLA: Towards Modeling of Object-Relation Representations in Robotic Manipulation
University of Arkansas（亚拉巴马大学）；FPT Software AI Center（FPT软件人工智能中心）；University of Stuttgart（斯图加特大学）；Aalborg University（奥尔堡大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Liverpool（利物浦大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）；Max Planck Research School for Intelligent Systems（马克斯·普朗克智能系统研究学校）
The ability to decompose complex natural scenes into meaningful object-centric abstractions lies at the core of human perception and reasoning. In the recent culmination of unsupervised object-centric learning, the Slot-Attention module has played an important role with its simple yet effective design and fostered many powerful variants. These methods, however, have been exceedingly difficult to train without supervision and are ambiguous in the notion of object, especially for complex natural scenes. In this paper, we propose to address these issues by investigating the potential of learnable…
Object-Centric Temporal Consistency via Conditional Autoregressive Inductive Biases
Delft University of Technology（代尔夫特理工大学）；The University of Tokyo（东京大学）；Mila, University of Montreal（蒙特利尔大学米拉研究所）；Google DeepMind（谷歌DeepMind）；Agency for Science, Technology and Research（新加坡科技研究局）；Nanyang Technological University（南洋理工大学）
We present a slot-wise, object-based transition model that decomposes a scene into objects, aligns them (with respect to a slot-wise object memory) to maintain a consistent order across time, and predicts how those objects evolve over successive frames. The model is trained end-to-end without supervision using losses at the level of the object-structured representation rather than pixels. Thanks to its alignment module, the model deals properly with two issues that are not handled satisfactorily by other transition models, namely object persistence and object identity. We show that the combina…
