Scene Separation & Data Selection: Temporal Segmentation Algorithm for Real-Time Video Stream Analysis
Southwest Jiaotong University（西南交通大学）；University of Leeds（利兹大学）
Yuelin Xin, Zihan Zhou, Yuxuan Xia
备注：
5 pages, 4 figures, at IJCAI-ECAI 2022 workshop, First International Workshop on Spatio-Temporal Reasoning and Learning, July 24, 2022, Vienna, Austria
我们提出了2SDS（Scene Separation and Data Selection algorithm），一种用于实时视频流解释的时间分割算法。它补充了基于CNN的模型，以利用视频中的时间信息。2SDS通过比较两帧之间的图像差异来检测视频流中场景之间的变化。它将视频分离成片段（场景），并且通过与CNN模型结合，2SDS能够为每个场景选择最优结果。在本文中，我们将讨论2SDS背后的一些基本方法和概念，并展示关于2SDS的一些初步实验结果。在这些实验中，2SDS实现了超过90%的总体准确率。
英文摘要：
We present 2SDS (Scene Separation and Data Selection algorithm), a temporal segmentation algorithm used in real-time video stream interpretation. It complements CNN-based models to make use of temporal information in videos. 2SDS can detect the change between scenes in a video stream by com-paring the image difference between two frames. It separates a video into segments (scenes), and by combining itself with a CNN model, 2SDS can select the optimal result for each scene. In this paper, we will be discussing some basic methods and concepts behind 2SDS, as well as presenting some preliminary experiment results regarding 2SDS. During these experiments, 2SDS has achieved an overall accuracy of over 90%.
University of Bonn（波恩大学）；RWTH Aachen University（亚琛工业大学）
Inferring the 3D geometry and the semantic meaning of surfaces, which are occluded, is a very challenging task. Recently, a first end-to-end learning approach has been proposed that completes a scene from a single depth image. The approach voxelizes the scene and predicts for each voxel if it is occupied and, if it is occupied, the semantic class label. In this work, we propose a two stream approach that leverages depth information and semantic information, which is inferred from the RGB image, for this task. The approach constructs an incomplete 3D semantic tensor, which uses a compact three-…
National Taiwan University（国立台湾大学）；Mobile Drive Technology
Nowadays, the need for user editing in a 3D scene has rapidly increased due to the development of AR and VR technology. However, the existing 3D scene completion task (and datasets) cannot suit the need because the missing regions in scenes are generated by the sensor limitation or object occlusion. Thus, we present a novel task named free-form 3D scene inpainting. Unlike scenes in previous 3D completion datasets preserving most of the main structures and hints of detailed shapes around missing regions, the proposed inpainting dataset, FF-Matterport, contains large and diverse missing regions …
Learning to Robustly Reconstruct Low-light Dynamic Scenes from Spike Streams
School of Computer Science, Peking University（北京大学计算机学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；College of Future Technology, Peking University（北京大学未来技术学院）
Peking University（北京大学）；Chinese University of Hong Kong（香港中文大学）
We revisit Semantic Scene Completion (SSC), a useful task to predict the semantic and occupancy representation of 3D scenes, in this paper. A number of methods for this task are always based on voxelized scene representations for keeping local scene structure. However, due to the existence of visible empty voxels, these methods always suffer from heavy computation redundancy when the network goes deeper, and thus limit the completion quality. To address this dilemma, we propose our novel point-voxel aggregation network for this task. Firstly, we transfer the voxelized scenes to point clouds by…
GSStream: 3D Gaussian Splatting based Volumetric Scene Streaming System
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）
Electrical and Computer Engineering and ASRI, Seoul National University（首尔国立大学电气与计算机工程系及自动化系统研究所）；Electrical Engineering and Computer Science, University of California at Merced（加州大学默塞德分校电气工程与计算机科学系）
Single image reflection separation is an ill-posed problem since two scenes, a transmitted scene and a reflected scene, need to be inferred from a single observation. To make the problem tractable, in this work we assume that categories of two scenes are known. It allows us to address the problem by generating both scenes that belong to the categories while their contents are constrained to match with the observed image. A novel network architecture is proposed to render realistic images of both scenes based on adversarial learning. The network can be trained in a weakly supervised manner, i.e…
University of Groningen（格罗宁根大学）；University of Barcelona（巴塞罗那大学）；University of Rovira i Virgili（罗维拉-威尔吉利大学）
Recent studies have shown that the environment where people eat can affect their nutritional behaviour. In this work, we provide automatic tools for a personalised analysis of a person's health habits by the examination of daily recorded egocentric photo-streams. Specifically, we propose a new automatic approach for the classification of food-related environments, that is able to classify up to 15 such scenes. In this way, people can monitor the context around their food intake in order to get an objective insight into their daily eating routine. We propose a model that classifies food-related…
University of Delaware（特拉华大学）；ShanghaiTech University（上海科技大学）；Microsoft Research（微软研究院）
We propose a novel approach that jointly removes reflection or translucent layer from a scene and estimates scene depth. The input data are captured via light field imaging. The problem is couched as minimizing the rank of the transmitted scene layer via Robust Principle Component Analysis (RPCA). We also impose regularization based on piecewise smoothness, gradient sparsity, and layer independence to simultaneously recover 3D geometry of the transmitted layer. Experimental results on synthetic and real data show that our technique is robust and reliable, and can handle a broad range of layer …
ClipGStream: Clip-Stream Gaussian Splatting for Any Length and Any Motion Multi-View Dynamic Scene Reconstruction
Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology（广东超高清沉浸式媒体技术省重点实验室）；Shenzhen Graduate School, Peking University（北京大学深圳研究生院）；Pengcheng Laboratory（鹏城实验室）；Peking University（北京大学）；MIGU Video Co., Ltd.（MIGU视频有限公司）
In this paper we propose an end-to-end swift 3D feature reductionist framework (3DFR) for scene independent change detection. The 3DFR framework consists of three feature streams: a swift 3D feature reductionist stream (AvFeat), a contemporary feature stream (ConFeat) and a temporal median feature map. These multilateral foreground/background features are further refined through an encoder-decoder network. As a result, the proposed framework not only detects temporal changes but also learns high-level appearance features. Thus, it incorporates the object semantics for effective change detectio…
Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video Understanding
Northwestern Polytechnical University（北华大学）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology（香港科技大学）；Harbin Engineering University（哈尔滨工程大学）
Peking University（北京大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
As a bio-inspired sensor with high temporal resolution, the spiking camera has an enormous potential in real applications, especially for motion estimation in high-speed scenes. However, frame-based and event-based methods are not well suited to spike streams from the spiking camera due to the different data modalities. To this end, we present, SCFlow, a tailored deep learning pipeline to estimate optical flow in high-speed scenes from spike streams. Importantly, a novel input representation is introduced which can adaptively remove the motion blur in spike streams according to the prior motio…
In this paper, we address the problem of reflection removal and deblurring from a single image captured by a plenoptic camera. We develop a two-stage approach to recover the scene depth and high resolution textures of the reflected and transmitted layers. For depth estimation in the presence of reflections, we train a classifier through convolutional neural networks. For recovering high resolution textures, we assume that the scene is composed of planar regions and perform the reconstruction of each layer by using an explicit form of the plenoptic camera point spread function. The proposed fra…
Real-world lighting often consists of multiple illuminants with different spectra. Separating and manipulating these illuminants in post-process is a challenging problem that requires either significant manual input or calibrated scene geometry and lighting. In this work, we leverage a flash/no-flash image pair to analyze and edit scene illuminants based on their spectral differences. We derive a novel physics-based relationship between color variations in the observed flash/no-flash intensities and the spectra and surface shading corresponding to individual scene illuminants. Our technique us…
We propose a framework for aligning and fusing multiple images into a single view using neural image representations (NIRs), also known as implicit or coordinate-based neural representations. Our framework targets burst images that exhibit camera ego motion and potential changes in the scene. We describe different strategies for alignment depending on the nature of the scene motion -- namely, perspective planar (i.e., homography), optical flow with minimal scene change, and optical flow with notable occlusion and disocclusion. With the neural image representation, our framework effectively com…
In this work we tackle the task of video-based visual emotion recognition in the wild. Standard methodologies that rely solely on the extraction of bodily and facial features often fall short of accurate emotion prediction in cases where the aforementioned sources of affective information are inaccessible due to head/body orientation, low resolution and poor illumination. We aspire to alleviate this problem by leveraging visual context in the form of scene characteristics and attributes, as part of a broader emotion recognition framework. Temporal Segment Networks (TSN) constitute the backbone…
Visual Computing Institute RWTH Aachen University（亚琛工业大学视觉计算研究所）
Semantic image segmentation is an essential component of modern autonomous driving systems, as an accurate understanding of the surrounding scene is crucial to navigation and action planning. Current state-of-the-art approaches in semantic image segmentation rely on pre-trained networks that were initially developed for classifying images as a whole. While these networks exhibit outstanding recognition performance (i.e., what is visible?), they lack localization accuracy (i.e., where precisely is something located?). Therefore, additional processing steps have to be performed in order to obtai…
HiCoM: Hierarchical Coherent Motion for Streamable Dynamic Scene with 3D Gaussian Splatting
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Pengcheng Laboratory（鹏城实验室）；Peking University Shenzhen Graduate School（北京大学深圳研究生院）
Single image reflection separation (SIRS), as a representative blind source separation task, aims to recover two layers, $\textit{i.e.}$, transmission and reflection, from one mixed observation, which is challenging due to the highly ill-posed nature. Existing deep learning based solutions typically restore the target layers individually, or with some concerns at the end of the output, barely taking into account the interaction across the two streams/branches. In order to utilize information more efficiently, this work presents a general yet simple interactive strategy, namely $\textit{your tr…
Two-Stream Interactive Joint Learning of Scene Parsing and Geometric Vision Tasks
College of Electronic and Information Engineering, Tongji University（同济大学电子信息学院）；Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学智能自主系统研究所）；College of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；Department of Vehicle Control System and Software Development, NIO（蔚来汽车车辆控制系统与软件开发部门）；Department of Automotive Engineering, Jilin University（吉林大学汽车工程学院）；Key Laboratory of Embedded System and Service Computing (Ministry of Education), Tongji University（同济大学嵌入式系统与服务计算重点实验室）；College of Electronic and Information Engineering, Shanghai Institute of Intelligent Science and Technology（上海智能科学与技术研究院电子信息学院）；Shanghai Key Laboratory of Intelligent Autonomous Systems（上海智能自主系统重点实验室）
StreamSplat: Towards Online Dynamic 3D Reconstruction from Uncalibrated Video Streams
University of British Columbia（不列颠哥伦比亚大学）；Vector Institute for AI（人工智能向量研究所）；Canada CIFAR AI Chair（加拿大CIFAR人工智能主席）；Nanyang Technological University（南洋理工大学）
Learning depth and ego-motion from unlabeled videos via self-supervision from epipolar projection can improve the robustness and accuracy of the 3D perception and localization of vision-based robots. However, the rigid projection computed by ego-motion cannot represent all scene points, such as points on moving objects, leading to false guidance in these regions. To address this problem, we propose an Attentional Separation-and-Aggregation Network (ASANet), which can learn to distinguish and extract the scene's static and dynamic characteristics via the attention mechanism. We further propose …
Scene understanding requires the extraction and representation of scene components together with their properties and inter-relations. We describe a model in which meaningful scene structures are extracted from the image by an iterative process, combining bottom-up (BU) and top-down (TD) networks, interacting through a symmetric bi-directional communication between them (counter-streams structure). The model constructs a scene representation by the iterative use of three components. The first model component is a BU stream that extracts selected scene elements, properties and relations. The se…
We propose a geometric superstructure for unified real-time processing of RGB-D data. Modern RGB-D sensors are widely used for indoor 3D capture, with applications ranging from modeling to robotics, through augmented reality. Nevertheless, their use is limited by their low resolution, with frames often corrupted with noise, missing data and temporal inconsistencies. Our approach consists in generating and updating through time a single set of compact local statistics parameterized over detected geometric proxies, which are fed from raw RGB-D data. Our proxies provide several processing primiti…
Attention-Guided Dual-Stream Learning for Group Engagement Recognition: Fusing Transformer-Encoded Motion Dynamics with Scene Context via Adaptive Gating
PAS3R: Pose-Adaptive Streaming 3D Reconstruction for Long Video Sequences
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University, China（人机混合增强智能国家重点实验室，人工智能与机器人研究院，西安交通大学，中国）；University of East Anglia, Norwich, NR47TJ, United Kingdom（东安格利亚大学，诺里奇，英国）
