Scalable Self-Supervised Representation Learning from Spatiotemporal Motion Trajectories for Multimodal Computer Vision
Apple（苹果公司）
Swetava Ganguli, C. V. Krishnakumar Iyer, Vipul Pandey
备注：
Extended abstract accepted for presentation at BayLearn 2022. 3 pages, 2 figures, 1 table. Abstract based on IEEE MDM 2022 research track paper: arXiv:2110.12521
Self-supervised representation learning techniques utilize large datasets without semantic annotations to learn meaningful, universal features that can be conveniently transferred to solve a wide variety of downstream supervised tasks. In this work, we propose a self-supervised method for learning representations of geographic locations from unlabeled GPS trajectories to solve downstream geospatial computer vision tasks. Tiles resulting from a raster representation of the earth's surface are modeled as nodes on a graph or pixels of an image. GPS trajectories are modeled as allowed Markovian paths on these nodes. A scalable and distributed algorithm is presented to compute image-like representations, called reachability summaries, of the spatial connectivity patterns between tiles and their neighbors implied by the observed Markovian paths. A convolutional, contractive autoencoder is trained to learn compressed representations, called reachability embeddings, of reachability summaries for every tile. Reachability embeddings serve as task-agnostic, feature representations of geographic locations. Using reachability embeddings as pixel representations for five different downstream geospatial tasks, cast as supervised semantic segmentation problems, we quantitatively demonstrate that reachability embeddings are semantically meaningful representations and result in 4-23% gain in performance, as measured using area under the precision-recall curve (AUPRC) metric, when compared to baseline models that use pixel representations that do not account for the spatial connectivity between tiles. Reachability embeddings transform sequential, spatiotemporal mobility data into semantically meaningful tensor representations that can be combined with other sources of imagery and are designed to facilitate multimodal learning in geospatial computer vision.
Self-supervised representation learning techniques utilize large datasets without semantic annotations to learn meaningful, universal features that can be conveniently transferred to solve a wide variety of downstream supervised tasks. In this paper, we propose a self-supervised method for learning representations of geographic locations from unlabeled GPS trajectories to solve downstream geospatial computer vision tasks. Tiles resulting from a raster representation of the earth's surface are modeled as nodes on a graph or pixels of an image. GPS trajectories are modeled as allowed Markovian p…
Shirley Ryan AbilityLab（雪莉·瑞安能力实验室）；Northwestern University（西北大学）；Baylor College of Medicine（贝勒医学院）
Human pose estimation from monocular video is a rapidly advancing field that offers great promise to human movement science and rehabilitation. This potential is tempered by the smaller body of work ensuring the outputs are clinically meaningful and properly calibrated. Gait analysis, typically performed in a dedicated lab, produces precise measurements including kinematics and step timing. Using over 7000 monocular video from an instrumented gait analysis lab, we trained a neural network to map 3D joint trajectories and the height of individuals onto interpretable biomechanical outputs includ…
Spatiotemporal-Augmented Graph Neural Networks for Human Mobility Simulation
College of Computer Science, Zhejiang University（浙江大学计算机学院）；College of Software Technology, Zhejiang University（浙江大学软件技术学院）；State Key Laboratory of Blockchain and Security, Zhejiang University（浙江大学区块链与安全国家重点实验室）
TU Darmstadt（达姆施塔特工业大学）；German Research Center for AI (DFKI)（德国人工智能研究中心（DFKI））；Centre for Cognitive Science, TU Darmstadt（达姆施塔特工业大学认知科学中心）
Modeling interaction dynamics to generate robot trajectories that enable a robot to adapt and react to a human's actions and intentions is critical for efficient and effective collaborative Human-Robot Interactions (HRI). Learning from Demonstration (LfD) methods from Human-Human Interactions (HHI) have shown promising results, especially when coupled with representation learning techniques. However, such methods for learning HRI either do not scale well to high dimensional data or cannot accurately adapt to changing via-poses of the interacting partner. We propose Multimodal Interactive Laten…
This paper describes a network that captures multimodal correlations over arbitrary timestamps. The proposed scheme operates as a complementary, extended network over a multimodal convolutional neural network (CNN). Spatial and temporal streams are required for action recognition by a deep CNN, but overfitting reduction and fusing these two streams remain open problems. The existing fusion approach averages the two streams. Here we propose a correlation network with a Shannon fusion for learning a pre-trained CNN. A Long-range video may consist of spatiotemporal correlations over arbitrary tim…
3DGS-Calib: 3D Gaussian Splatting for Multimodal SpatioTemporal Calibration
Huawei Paris Research Center（华为巴黎研究中心）；Noah’s Ark（诺亚方舟）；ICB UMR CNRS 6303, Université de Bourgogne（勃艮第大学ICB UMR CNRS 6303研究所）；Huawei London Research Center（华为伦敦研究中心）；ImViA UR 7535, Université de Bourgogne（勃艮第大学ImViA UR 7535研究所）；MIS UR 4290, Université de Picardie Jules Verne（儒勒·凡尔纳皮卡第大学MIS UR 4290研究所）
ChangeBridge: Spatiotemporal Image Generation with Multimodal Controls for Remote Sensing
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；Zhongguancun Academy（中关村学院）；Graduate School of Frontier Sciences, The University of Tokyo（东京大学新领域创成科学研究科）；Department of Computer Science, Stanford University（斯坦福大学计算机科学系）
LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Models
Institute of Image Communication and Information Processing, Shanghai Jiao Tong University（上海交通大学图像通信与信息处理研究所）；MoE Key Laboratory of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院（教育部人工智能重点实验室））；Huawei Technologies（华为技术有限公司）
IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learning
Hangzhou International Innovation Institute, Beihang University（北京航空航天大学杭州国际创新研究院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Kuaishou Technology（快手科技）；Shenzhen Institute of Advanced Integration Technology, Shenzhen（深圳先进集成技术研究院）
GoalFlow: Goal-Driven Flow Matching for Multimodal Trajectories Generation in End-to-End Autonomous Driving
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Horizon Robotics；Nanjing University（南京大学）；Huazhong University of Science & Technology（华中科技大学）；Shanghai AI Laboratory（上海人工智能实验室）
Cornell University（康奈尔大学）；Stanford University（斯坦福大学）；Brain Of Things, Inc.（Brain Of Things 公司）
A robot operating in a real-world environment needs to perform reasoning over a variety of sensor modalities such as vision, language and motion trajectories. However, it is extremely challenging to manually design features relating such disparate modalities. In this work, we introduce an algorithm that learns to embed point-cloud, natural language, and manipulation trajectory data into a shared embedding space with a deep neural network. To learn semantically meaningful spaces throughout our network, we use a loss-based margin to bring embeddings of relevant pairs closer together while drivin…
In Argoverse motion forecasting competition, the task is to predict the probabilistic future trajectory distribution for the interested targets in the traffic scene. We use vectorized lane map and 2 s targets' history trajectories as input. Then the model outputs 6 forecasted trajectories with probability for each target.
The Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
Bundle adjustment jointly optimizes camera intrinsics and extrinsics and 3D point triangulation to reconstruct a static scene. The triangulation constraint, however, is invalid for moving points captured in multiple unsynchronized videos and bundle adjustment is not designed to estimate the temporal alignment between cameras. We present a spatiotemporal bundle adjustment framework that jointly optimizes four coupled sub-problems: estimating camera intrinsics and extrinsics, triangulating static 3D points, as well as sub-frame temporal alignment between cameras and computing 3D trajectories of …
The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）
Predicting multiple plausible future trajectories of the nearby vehicles is crucial for the safety of autonomous driving. Recent motion prediction approaches attempt to achieve such multimodal motion prediction by implicitly regularizing the feature or explicitly generating multiple candidate proposals. However, it remains challenging since the latent features may concentrate on the most frequent mode of the data while the proposal-based methods depend largely on the prior knowledge to generate and select the proposals. In this work, we propose a novel transformer framework for multimodal moti…
Beijing Lab of Intelligent Information Technology, School of Computer Science, Beijing Institute of Technology（北京理工大学计算机学院智能信息技术北京市重点实验室）；University of California, Los Angeles（加州大学洛杉矶分校）；Research School of Engineering, Australian National University（澳大利亚国立大学工程研究院）
We introduce a novel semi-supervised video segmentation approach based on an efficient video representation, called as "super-trajectory". Each super-trajectory corresponds to a group of compact trajectories that exhibit consistent motion patterns, similar appearance and close spatiotemporal relationships. We generate trajectories using a probabilistic model, which handles occlusions and drifts in a robust and natural way. To reliably group trajectories, we adopt a modified version of the density peaks based clustering algorithm that allows capturing rich spatiotemporal relations among traject…
Beijing University of Posts and Telecommunications（北京邮电大学）
Predicting the plausible future trajectories of nearby agents is a core challenge for the safety of Autonomous Vehicles and it mainly depends on two external cues: the dynamic neighbor agents and static scene context. Recent approaches have made great progress in characterizing the two cues separately. However, they ignore the correlation between the two cues and most of them are difficult to achieve map-adaptive prediction. In this paper, we use lane as scene data and propose a staged network that Jointly learning Agent and Lane information for Multimodal Trajectory Prediction (JAL-MTP). JAL-…
LongFly: Long-Horizon UAV Vision-and-Language Navigation with Spatiotemporal Context Integration
School of Mechanical Engineering, Beijing Institute of Technology（北京理工大学机械工程学院）；Chongqing Innovation Center, Beijing Institute of Technology（北京理工大学重庆创新中心）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Department of Automation, Tsinghua University（清华大学自动化系）；School of Software, Dalian University of Technology（大连理工大学软件学院）；School of Artificial Intelligence, Xidian University（西安电子科技大学人工智能学院）
Semi-disentangled spatiotemporal implicit neural representations of longitudinal neuroimaging data for trajectory classification
Biomedical Engineering Graduate Program, University of Calgary, Calgary, Canada（卡利告大学生物医学工程研究生项目）；Department of Radiology, University of Calgary, Calgary, Canada（卡利告大学放射科）；Department of Radiology, University of Michigan, Ann Arbor, United States（密歇根大学放射科）
What, Where, and How: Probing Spatiotemporal Representations in Video Foundation Models
Lassonde School of Engineering（拉桑德工程学院）；York University（约克大学）；Vector Institute（矢量研究院）；FAIR, Meta Superintelligence Labs（Meta超级智能实验室FAIR）；Mila（米拉研究院）；McGill University（麦吉尔大学）；Goodfire AI（Goodfire人工智能公司）
VLA-4D: Embedding 4D Awareness into Vision-Language-Action Models for SpatioTemporally Coherent Robotic Manipulation
School of Computing, National University of Singapore（新加坡国立大学计算机学院）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）
Multimodal Diffusion Transformer with Memory Bank for Scalable Long-Duration Talking Video Generation
South China University of Technology（南方科技大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing National Research Center for Information Science and Technology, Tsinghua University（北京信息科学研究中心，清华大学）；Department of Automation, BNRist, Tsinghua University（清华大学自动化系，北京信息科学研究中心）
University of Edinburgh（爱丁堡大学）；Google Research（谷歌研究院）
Given unstructured videos of deformable objects, we automatically recover spatiotemporal correspondences to map one object to another (such as animals in the wild). While traditional methods based on appearance fail in such challenging conditions, we exploit consistency in object motion between instances. Our approach discovers pairs of short video intervals where the object moves in a consistent manner and uses these candidates as seeds for spatial alignment. We model the spatial correspondence between the point trajectories on the object in one interval to those in the other using a time-var…
Hanyang University（汉阳大学）；Yonsei University（延世大学）；Carnegie Mellon University（卡内基梅隆大学）；Korea University（高丽大学）；Sogang University（西江大学）；Bosch Research Pittsburgh（博世匹兹堡研究中心）
Multi-agent trajectory forecasting in autonomous driving requires an agent to accurately anticipate the behaviors of the surrounding vehicles and pedestrians, for safe and reliable decision-making. Due to partial observability in these dynamical scenes, directly obtaining the posterior distribution over future agent trajectories remains a challenging problem. In realistic embodied environments, each agent's future trajectories should be both diverse since multiple plausible sequences of actions can be used to reach its intended goals, and admissible since they must obey physical constraints an…
Predicting the possible future trajectories of the surrounding dynamic agents is an essential requirement in autonomous driving. These trajectories mainly depend on the surrounding static environment, as well as the past movements of those dynamic agents. Furthermore, the multimodal nature of agent intentions makes the trajectory prediction problem more challenging. All of the existing models consider the target agent as well as the surrounding agents similarly, without considering the variation of physical properties. In this paper, we present a novel deep-learning based framework for multimo…
