Whole understanding of the surroundings is paramount to autonomous systems. Recent works have shown that deep neural networks can learn geometry (depth) and motion (optical flow) from a monocular video without any explicit supervision from ground truth annotations, particularly hard to source for these two tasks. In this paper, we take an additional step toward holistic scene understanding with monocular cameras by learning depth and motion alongside with semantics, with supervision for the latter provided by a pre-trained network distilling proxy ground truth images. We address the three tasks jointly by a) a novel training protocol based on knowledge distillation and self-supervision and b) a compact network architecture which enables efficient scene understanding on both power hungry GPUs and low-power embedded platforms. We thoroughly assess the performance of our framework and show that it yields state-of-the-art results for monocular depth estimation, optical flow and motion segmentation.
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；RPTU University Kaiserslautern-Landau（科布伦茨-兰道大学（RPTU））；University of Modena and Reggio Emilia（摩德纳和雷吉奥-艾米利亚大学）
SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splatting
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；ETH Zürich（苏黎世联邦理工学院）；University of Amsterdam（阿姆斯特丹大学）；Johns Hopkins University（约翰·霍普金斯大学）；University of Pisa（比萨大学）；University of Trento（特伦托大学）
University of Toronto（多伦多大学）；Vector Institute（矢量研究院）；UBC（不列颠哥伦比亚大学）；Google Research（谷歌研究院）；NVIDIA（英伟达）；SFU（西蒙弗雷泽大学）
We introduce a technique for pairwise registration of neural fields that extends classical optimization-based local registration (i.e. ICP) to operate on Neural Radiance Fields (NeRF) -- neural 3D scene representations trained from collections of calibrated images. NeRF does not decompose illumination and color, so to make registration invariant to illumination, we introduce the concept of a ''surface field'' -- a field distilled from a pre-trained NeRF model that measures the likelihood of a point being on the surface of an object. We then cast nerf2nerf registration as a robust optimization …
Since visual perception can give rich information beyond text descriptions for world understanding, there has been increasing interest in leveraging visual grounding for language learning. Recently, vokenization (Tan and Bansal, 2020) has attracted attention by using the predictions of a text-to-image retrieval model as labels for language model supervision. Despite its success, the method suffers from approximation error of using finite image labels and the lack of vocabulary diversity of a small image-text dataset. To overcome these limitations, we present VidLanKD, a video-language knowledg…
Preferred Networks, Inc.（首选网络公司）；Massachusetts Institute of Technology（麻省理工学院）
Emerging neural radiance fields (NeRF) are a promising scene representation for computer graphics, enabling high-quality 3D reconstruction and novel view synthesis from image observations. However, editing a scene represented by a NeRF is challenging, as the underlying connectionist representations such as MLPs or voxel grids are not object-centric or compositional. In particular, it has been difficult to selectively edit specific regions or objects. In this work, we tackle the problem of semantic scene decomposition of NeRFs to enable query-based local editing of the represented 3D scenes. We…
Physics-Distilled Neural Network enabled by Large Language Models for Manufacturing Process-Property Predictive Modeling
School of Mechanical, Aerospace and Manufacturing Engineering, University of Connecticut（康奈尔大学机械、航空航天与制造工程学院）；Department of Mechanical & Aerospace Engineering, Rutgers, the State University of New Jersey（新泽西州立大学鲁特大学机械与航空航天工程学院）
We propose a method to infer a dense depth map from a single image, its calibration, and the associated sparse point cloud. In order to leverage existing models (teachers) that produce putative depth maps, we propose an adaptive knowledge distillation approach that yields a positive congruent training process, wherein a student model avoids learning the error modes of the teachers. In the absence of ground truth for model selection and training, our method, termed Monitored Distillation, allows a student to exploit a blind ensemble of teachers by selectively learning from predictions that best…
A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenes
Hangzhou Normal University（杭州师范大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
MM-3DScene: 3D Scene Understanding by Customizing Masked Modeling with Informative-Preserved Reconstruction and Self-Distilled Consistency
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Tecnologico de Monterrey（蒙特雷科技大学）；University of Leeds（利兹大学）
Minimally invasive surgery is highly operator dependant with a lengthy procedural time causing fatigue to surgeon and risks to patients such as injury to organs, infection, bleeding, and complications of anesthesia. To mitigate such risks, real-time systems are desired to be developed that can provide intra-operative guidance to surgeons. For example, an automated system for tool localization, tool (or tissue) tracking, and depth estimation can enable a clear understanding of surgical scenes preventing miscalculations during surgical procedures. In this work, we present a systematic review of …
TaxDistill: Improving Metagenomic Taxonomic Annotation via Distilled Genomic Foundation Models
National Genomics Data Center, China National Center for Bioinformation（中国生物信息中心国家基因组数据中心）；Beijing Key Laboratory of Intelligent Governance and Application of Biological Big Data, China National Center for Bioinformation（北京生物大数据智能治理与应用重点实验室，中国生物信息中心）；Beijing Institute of Genomics, Chinese Academy of Sciences（北京基因组研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）
We present MAV3D (Make-A-Video3D), a method for generating three-dimensional dynamic scenes from text descriptions. Our approach uses a 4D dynamic Neural Radiance Field (NeRF), which is optimized for scene appearance, density, and motion consistency by querying a Text-to-Video (T2V) diffusion-based model. The dynamic video output generated from the provided text can be viewed from any camera location and angle, and can be composited into any 3D environment. MAV3D does not require any 3D or 4D data and the T2V model is trained only on Text-Image pairs and unlabeled videos. We demonstrate the ef…
DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features
NVIDIA Research（英伟达研究院）；University of Toronto（多伦多大学）；University of Southern California（南加州大学）；National University of Singapore（新加坡国立大学）；Stanford University（斯坦福大学）
Point Cloud Mixture-of-Domain-Experts Model for 3D Self-supervised Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；Institute of Perceptual Intelligence, Pengcheng Laboratory（感知智能研究院，鹏城实验室）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳学院）
ETH Zürich（苏黎世联邦理工学院）；University of Würzburg（维尔茨堡大学）
NeRFmm is the Neural Radiance Fields (NeRF) that deal with Joint Optimization tasks, i.e., reconstructing real-world scenes and registering camera parameters simultaneously. Despite NeRFmm producing precise scene synthesis and pose estimations, it still struggles to outperform the full-annotated baseline on challenging scenes. In this work, we identify that there exists a systematic sub-optimality in joint optimization and further identify multiple potential sources for it. To diminish the impacts of potential sources, we propose Sinusoidal Neural Radiance Fields (SiNeRF) that leverage sinusoi…
FocusDD: Real-World Scene Infusion for Robust Dataset Distillation
Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；Swiss Data Science Center（瑞士数据科学中心）；Graz University of Technology（格拉茨工业大学）；Complexity Science Hub Vienna（维也纳复杂科学中心）
Oxford Robotics Institute（牛津机器人研究所）；University of Oxford（牛津大学）
This work explores scene graphs as a distilled representation of high-level information for autonomous driving, applied to future driver-action prediction. Given the scarcity and strong imbalance of data samples, we propose a self-supervision pipeline to infer representative and well-separated embeddings. Key aspects are interpretability and explainability; as such, we embed in our architecture attention mechanisms that can create spatial and temporal heatmaps on the scene graphs. We evaluate our system on the ROAD dataset against a fully-supervised approach, showing the superiority of our tra…
Self-Distilled Masked Auto-Encoders are Efficient Video Anomaly Detectors
University of Bucharest（布加勒斯特大学）；NUST Politehnica Bucharest（布加勒斯特理工大学）；SecurifAI（SecurifAI公司）；MBZ University of Artificial Intelligence（MBZ人工智能大学）；Linköping University（林雪平大学）；University of Central Florida（中佛罗里达大学）
Insight Centre for Data Analytics, Dublin City University（都柏林城市大学洞察数据分析中心）
As part of the MediaEval 2022 Predicting Video Memorability task we explore the relationship between visual memorability, the visual representation that characterises it, and the underlying concept portrayed by that visual representation. We achieve state-of-the-art memorability prediction performance with a model trained and tested exclusively on surrogate dream images, elevating concepts to the status of a cornerstone memorability feature, and finding strong evidence to suggest that the intrinsic memorability of visual content can be distilled to its underlying concept or meaning irrespectiv…
University of Technology Sydney（悉尼科技大学）；Northwest University（西北大学）；Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；Shandong Artificial Intelligence Institute（山东省人工智能研究院）；Carnegie Mellon University（卡内基梅隆大学）
Scene graph is a structured representation of a scene that can clearly express the objects, attributes, and relationships between objects in the scene. As computer vision technology continues to develop, people are no longer satisfied with simply detecting and recognizing objects in images; instead, people look forward to a higher level of understanding and reasoning about visual scenes. For example, given an image, we want to not only detect and recognize objects in the image, but also know the relationship between objects (visual relationship detection), and generate a text description (imag…
Xidian University（西安电子科技大学）；National University of Defense Technology（国防科技大学）；Edith Cowan University（埃迪斯科文大学）；The University of Western Australia（西澳大学）
Deep learning techniques have led to remarkable breakthroughs in the field of generic object detection and have spawned a lot of scene-understanding tasks in recent years. Scene graph has been the focus of research because of its powerful semantic representation and applications to scene understanding. Scene Graph Generation (SGG) refers to the task of automatically mapping an image into a semantic structural scene graph, which requires the correct labeling of detected objects and their relationships. Although this is a challenging task, the community has proposed a lot of SGG approaches and a…
H2G: Hierarchy-Aware Hyperbolic Grouping for 3D Scenes
Department of Computer Science and Engineering, Korea University（韩国大学计算机科学与工程系）；Intelligence and Interaction Research Center, Korea Institute of Science and Technology（韩国科学技术院智能与交互研究中心）
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Tencent AI Lab（腾讯人工智能实验室）
We address the challenging problem of image captioning by revisiting the representation of image scene graph. At the core of our method lies the decomposition of a scene graph into a set of sub-graphs, with each sub-graph capturing a semantic component of the input image. We design a deep model to select important sub-graphs, and to decode each selected sub-graph into a single target sentence. By using sub-graphs, our model is able to attend to different components of the image. Our method thus accounts for accurate, diverse, grounded and controllable captioning at the same time. We present ex…
Huazhong University of Science and Technology（华中科技大学）
Ethiopic/Amharic script is one of the oldest African writing systems, which serves at least 23 languages (e.g., Amharic, Tigrinya) in East Africa for more than 120 million people. The Amharic writing system, Abugida, has 282 syllables, 15 punctuation marks, and 20 numerals. The Amharic syllabic matrix is derived from 34 base graphemes/consonants by adding up to 12 appropriate diacritics or vocalic markers to the characters. The syllables with a common consonant or vocalic markers are likely to be visually similar and challenge text recognition tasks. In this work, we presented the first compre…
CVIT（计算机视觉与智能系统中心）；Kohli Center on Intelligent Systems (KCIS)（科利智能系统中心）；IIIT Hyderabad（海得拉巴国际信息技术学院）
Radiance Fields (RF) are popular to represent casually-captured scenes for new view synthesis and several applications beyond it. Mixed reality on personal spaces needs understanding and manipulating scenes represented as RFs, with semantic segmentation of objects as an important step. Prior segmentation efforts show promise but don't scale to complex objects with diverse appearance. We present the ISRF method to interactively segment objects with fine structure and appearance. Nearest neighbor feature matching using distilled semantic features identifies high-confidence seed regions. Bilatera…
Comprehensive Visual Question Answering on Point Clouds through Compositional Scene Manipulation
The Future Network of Intelligence Institute, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）未来智能网络研究院）；School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）理工学院）
Brain-Semantoks: Learning Semantic Tokens of Brain Dynamics with a Self-Distilled Foundation Model
Hertie Institute for AI in Brain Health, University of Tübingen（图宾根大学脑健康人工智能研究所）；Tübingen AI Center, University of Tübingen（图宾根大学图宾根人工智能中心）；Charité – Universitätsmedizin Berlin, Department of Psychiatry and Psychotherapy（柏林夏里特医学院心理治疗系）
SceneScribe-1M: A Large-Scale Video Dataset with Comprehensive Geometric and Semantic Annotations
Shanghai Jiao Tong University（上海交通大学）；Ant Group（蚂蚁集团）；Visual Geometry Group, University of Oxford（牛津大学视觉几何组）；Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo（宁波数字孪生研究院，东部技术研究院，宁波）；Zhejiang Key Laboratory of Industrial Intelligence and Digital Twin（浙江工业智能与数字孪生重点实验室）
