This paper presents abstract art created by neural networks and broadly recognizable across various computer vision systems. The existence of abstract forms that trigger specific labels independent of neural architecture or training set suggests convolutional neural networks build shared visual representations for the categories they understand. Computer vision classifiers encountering these drawings often respond with strong responses for specific labels - in extreme cases stronger than all examples from the validation set. By surveying human subjects we confirm that these abstract artworks are also broadly recognizable by people, suggesting visual representations triggered by these drawings are shared across human and computer vision systems.
We introduce the Neural State Machine, seeking to bridge the gap between the neural and symbolic views of AI and integrate their complementary strengths for the task of visual reasoning. Given an image, we first predict a probabilistic graph that represents its underlying semantics and serves as a structured world model. Then, we perform sequential reasoning over the graph, iteratively traversing its nodes to answer a given question or draw a new inference. In contrast to most neural architectures that are designed to closely interact with the raw sensory data, our model operates instead in an…
Mila, Université de Montréal（米拉，蒙特利尔大学）；Kakao Brain；Rutgers University（罗格斯大学）
We introduce a variational approach to learning and inference of temporally hierarchical structure and representation for sequential data. We propose the Variational Temporal Abstraction (VTA), a hierarchical recurrent state space model that can infer the latent temporal structure and thus perform the stochastic state transition hierarchically. We also propose to apply this model to implement the jumpy-imagination ability in imagination-augmented agent-learning in order to improve the efficiency of the imagination. In experiments, we demonstrate that our proposed method can model 2D and 3D vis…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Tübingen（蒂宾根大学）；Boston University（波士顿大学）
It is well known that semantic segmentation can be used as an effective intermediate representation for learning driving policies. However, the task of street scene semantic segmentation requires expensive annotations. Furthermore, segmentation algorithms are often trained irrespective of the actual driving task, using auxiliary image-space loss functions which are not guaranteed to maximize driving metrics such as safety or distance traveled per intervention. In this work, we seek to quantify the impact of reducing segmentation annotation costs on learned behavior cloning agents. We analyze s…
People can produce drawings of specific entities (e.g., Garfield), as well as general categories (e.g., "cat"). What explains this ability to produce such varied drawings of even highly familiar object concepts? We hypothesized that drawing objects at different levels of abstraction depends on both sensory information and representational goals, such that drawings intended to portray a recently seen object preserve more detail than those intended to represent a category. Participants drew objects cued either with a photo or a category label. For each cue type, half the participants aimed to dr…
Aligning Machine and Human Visual Representations across Abstraction Levels
Google DeepMind Machine Learning Group（谷歌DeepMind机器学习组）；Technische Universität Berlin（技术大学柏林）；BIFOLD；Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）；Max Planck Institute for Human Cognitive and Brain Sciences（人类认知与脑科学Max Planck研究所）；Max Planck Institute for Human Development（人类发展Max Planck研究所）；TUD Dresden University of Technology（德累斯顿技术大学）；Anthropic；Department of Artificial Intelligence, Korea University（人工智能系，韩国大学）；Max Planck Institute for Informatics（信息Max Planck研究所）
Shenzhen University（深圳大学）；Visual Computing Research Center (VCC), College of Computer Science and Software Engineering (CSSE)（视觉计算研究中心（VCC），计算机科学与软件工程学院）；Peking University（北京大学）
提出Abstraction in Style (AiS)框架，将结构抽象与视觉风格分离，通过中间抽象代理实现几何保真度放松，从而支持更广泛的非真实感风格迁移。
University of California Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）；Massachusetts Institute of Technology（麻省理工学院）
This paper tests the hypothesis that modeling a scene in terms of entities and their local interactions, as opposed to modeling the scene globally, provides a significant benefit in generalizing to physical tasks in a combinatorial space the learner has not encountered before. We present object-centric perception, prediction, and planning (OP3), which to the best of our knowledge is the first fully probabilistic entity-centric dynamic latent variable framework for model-based reinforcement learning that acquires entity representations from raw visual observations without supervision and uses t…
SEVA: Leveraging sketches to evaluate alignment between human and machine visual abstraction
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of California, San Diego（加州大学圣地亚哥分校）；Tel-Aviv University（特拉维夫大学）；Reichman University（赖希曼大学）；Stanford University（斯坦福大学）
本文提出 SEVA 基准，用约 9 万张不同稀疏度的人类草图评估视觉算法的识别与人类响应对齐程度，并考察草图生成模型，揭示模型与人类仍有明显差距。
We present an investigation into how representational losses can affect the drawings produced by artificial agents playing a communication game. Building upon recent advances, we show that a combination of powerful pretrained encoder networks, with appropriate inductive biases, can lead to agents that draw recognisable sketches, whilst still communicating well. Further, we start to develop an approach to help automatically analyse the semantic content being conveyed by a sketch and demonstrate that current approaches to inducing perceptual biases lead to a notion of objectness being a key feat…
Humans routinely retrace paths in a novel environment both forwards and backwards despite uncertainty in their motion. This paper presents an approach for doing so. Given a demonstration of a path, a first network generates a path abstraction. Equipped with this abstraction, a second network observes the world and decides how to act to retrace the path under noisy actuation and a changing environment. The two networks are optimized end-to-end at training time. We evaluate the method in two realistic simulators, performing path following and homing under actuation noise and environmental change…
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；Austrian Institute of Technology（奥地利技术研究所）
We propose a new abstraction set (SynopSet) that has a continuum of visual representations for the explanatory analysis of molecular dynamics simulations (MDS) in the DNA nanotechnology domain. By re-purposing the commonly used progress bar and designing novel visuals, as well as transforming the data from the domain format to a format that better fits the newly designed visuals, we compose this new set of representations. This set is also designed to be capable of showing all spatial and temporal details, and all structural complexity, or abstracting these to various degrees, enabling both th…
Swiss Federal Institute of Technology (EPFL)（瑞士联邦理工学院（洛桑））；Tel-Aviv University（特拉维夫大学）；Reichman University（赖希曼大学）
Abstraction is at the heart of sketching due to the simple and minimal nature of line drawings. Abstraction entails identifying the essential visual properties of an object or scene, which requires semantic understanding and prior knowledge of high-level concepts. Abstract depictions are therefore challenging for artists, and even more so for machines. We present CLIPasso, an object sketching method that can achieve different levels of abstraction, guided by geometric and semantic simplifications. While sketch generation methods often rely on explicit sketch datasets for training, we utilize t…
VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning
University of Cambridge（剑桥大学）；Cornell University（康奈尔大学）；Massachusetts Institute of Technology（麻省理工学院）；Princeton University（普林斯顿大学）；University of Oxford（牛津大学）；The Alan Turing Institute（阿兰·图灵研究所）
UC Berkeley（加州大学伯克利分校）；Meta AI Research（Meta人工智能研究院）；Princeton University（普林斯顿大学）；UT Austin（德克萨斯大学奥斯汀分校）
Object rearrangement is a challenge for embodied agents because solving these tasks requires generalizing across a combinatorially large set of configurations of entities and their locations. Worse, the representations of these entities are unknown and must be inferred from sensory percepts. We present a hierarchical abstraction approach to uncover these underlying entities and achieve combinatorial generalization from unstructured visual inputs. By constructing a factorized transition graph over clusters of entity representations inferred from pixels, we show how to learn a correspondence bet…
Shortage of fully annotated datasets has been a limiting factor in developing deep learning based image segmentation algorithms and the problem becomes more pronounced in multi-organ segmentation. In this paper, we propose a unified training strategy that enables a novel multi-scale deep neural network to be trained on multiple partially labeled datasets for multi-organ segmentation. In addition, a new network architecture for multi-scale feature abstraction is proposed to integrate pyramid input and feature analysis into a U-shape pyramid structure. To bridge the semantic gap caused by direct…
We study open-world 3D scene understanding, a family of tasks that require agents to reason about their 3D environment with an open-set vocabulary and out-of-domain visual inputs - a critical skill for robots to operate in the unstructured 3D world. Towards this end, we propose Semantic Abstraction (SemAbs), a framework that equips 2D Vision-Language Models (VLMs) with new 3D spatial capabilities, while maintaining their zero-shot robustness. We achieve this abstraction using relevancy maps extracted from CLIP, and learn 3D spatial and geometric reasoning skills on top of those abstractions in…
Università degli Studi di Milano - Bicocca（米兰比可卡大学）；University of Freiburg（弗莱堡大学）；Universidad de Alcalá（阿尔卡拉大学）
Global localization is an important and widely studied problem for many robotic applications. Place recognition approaches can be exploited to solve this task, e.g., in the autonomous driving field. While most vision-based approaches match an image w.r.t. an image database, global visual localization within LiDAR-maps remains fairly unexplored, even though the path toward high definition 3D maps, produced mainly from LiDARs, is clear. In this work we leverage Deep Neural Network (DNN) approaches to create a shared embedding space between images and LiDAR-maps, allowing for image to 3D-LiDAR pl…
