We propose the Temporal Point Cloud Networks (TPCN), a novel and flexible framework with joint spatial and temporal learning for trajectory prediction. Unlike existing approaches that rasterize agents and map information as 2D images or operate in a graph representation, our approach extends ideas from point cloud learning with dynamic temporal learning to capture both spatial and temporal information by splitting trajectory prediction into both spatial and temporal dimensions. In the spatial dimension, agents can be viewed as an unordered point set, and thus it is straightforward to apply point cloud learning techniques to model agents' locations. While the spatial dimension does not take kinematic and motion information into account, we further propose dynamic temporal learning to model agents' motion over time. Experiments on the Argoverse motion forecasting benchmark show that our approach achieves the state-of-the-art results.
Talk2PC: Enhancing 3D Visual Grounding through LiDAR and Radar Point Clouds Fusion for Autonomous Driving
Thrust of Artificial Intelligence, Hong Kong University of Science and Technology (Guangzhou), China（香港理工大学（广州）人工智能研究所）；Mononai AI, Sweden（蒙诺人工智能）；College of Computer Science and Technology, China University of Petroleum (East China)（中国石油大学（华东）计算机科学与技术学院）；School of Advanced Technology, Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学先进科技学院）；Wangxuan Institute of Computer Technology, Peking University（北京大学王轩计算机技术研究所）；School of Automation, Southeast University（东南大学自动化学院）；College of Science and Engineering, James Cook University（詹姆斯库克大学科学与工程学院）；School of Transportation, Southeast University（东南大学交通运输学院）
The City College of New York（纽约城市学院）；The City College（城市学院）；the Graduate Center, the City University of New York（纽约城市大学研究生中心）
Point cloud has drawn more and more research attention as well as real-world applications. However, many of these applications (e.g. autonomous driving and robotic manipulation) are actually based on sequential point clouds (i.e. four dimensions) because the information of the static point cloud data could provide is still limited. Recently, researchers put more and more effort into sequential point clouds. This paper presents an extensive review of the deep learning-based methods for sequential point cloud research including dynamic flow estimation, object detection \& tracking, point cloud s…
Understanding 3D scenes is a critical prerequisite for autonomous agents. Recently, LiDAR and other sensors have made large amounts of data available in the form of temporal sequences of point cloud frames. In this work, we propose a novel problem -- sequential scene flow estimation (SSFE) -- that aims to predict 3D scene flow for all pairs of point clouds in a given sequence. This is unlike the previously studied problem of scene flow estimation which focuses on two frames. We introduce the SPCM-Net architecture, which solves this problem by computing multi-scale spatiotemporal correlations b…
Australian National University（澳大利亚国立大学）；EPFL（洛桑联邦理工学院）；ClearSpace
In this paper, we tackle the task of scene-aware 3D human motion forecasting, which consists of predicting future human poses given a 3D scene and a past human motion. A key challenge of this task is to ensure consistency between the human and the scene, accounting for human-scene interactions. Previous attempts to do so model such interactions only implicitly, and thus tend to produce artifacts such as "ghost motion" because of the lack of explicit constraints between the local poses and the global motion. Here, by contrast, we propose to explicitly model the human-scene contacts. To this end…
Global 3D Reconstruction of Clouds & Tropical Cyclones
University of Oxford（牛津大学）；Universitat de València（瓦伦西亚大学）；National Observatory of Athens（雅典国家天文台）；National Technical University of Athens（雅典技术大学）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；European Space Agency（欧洲航天局）；Environment and Climate Change Canada（加拿大环境与气候变化部）
Distributed solar generation forecasting using attention-based deep neural networks for cloud movement prediction
Department of Mechanical Engineering, School of Electrical, Mechanical and Infrastructure Engineering, The University of Melbourne, Melbourne, Australia（机械工程系，电气、机械与基础设施工程学院，墨尔本大学，墨尔本，澳大利亚）；Department of Mechanical Engineering, Sri Lanka Institute of Information Technology, Sri Lanka（机械工程系，斯里兰卡信息科技研究所，斯里兰卡）；School of Computing and Information Systems, Melbourne Centre for Data Science, The University of Melbourne, Melbourne, Australia（计算与信息学院，墨尔本数据科学中心，墨尔本大学，墨尔本，澳大利亚）
Stanford University（斯坦福大学）；University of Cambridge（剑桥大学）；Engie Lab CRIGEN（昂热集团CRIGEN实验室）；Universität Salzburg（萨尔茨堡大学）；Université Bretagne Sud（南布列塔尼大学）
Sky-image-based solar forecasting using deep learning has been recognized as a promising approach in reducing the uncertainty in solar power generation. However, one of the biggest challenges is the lack of massive and diversified sky image samples. In this study, we present a comprehensive survey of open-source ground-based sky image datasets for very short-term solar forecasting (i.e., forecasting horizon less than 30 minutes), as well as related research areas which can potentially help improve solar forecasting methods, including cloud segmentation, cloud classification and cloud motion pr…
The Pennsylvania State University（宾夕法尼亚州立大学）；Amazon Lab126（亚马逊Lab126）；Accuweather Inc.（Accuweather公司）；University of Almería（阿尔梅里亚大学）
Meteorologists use shapes and movements of clouds in satellite images as indicators of several major types of severe storms. Satellite imaginary data are in increasingly higher resolution, both spatially and temporally, making it impossible for humans to fully leverage the data in their forecast. Automatic satellite imagery analysis methods that can find storm-related cloud patterns as soon as they are detectable are in demand. We propose a machine learning and pattern recognition based approach to detect "comma-shaped" clouds in satellite images, which are specific cloud distribution patterns…
Forecasting the formation and development of clouds is a central element of modern weather forecasting systems. Incorrect clouds forecasts can lead to major uncertainty in the overall accuracy of weather forecasts due to their intrinsic role in the Earth's climate system. Few studies have tackled this challenging problem from a machine learning point-of-view due to a shortage of high-resolution datasets with many historical observations globally. In this paper, we present a novel satellite-based dataset called ``CloudCast''. It consists of 70,080 images with 10 different cloud types for multip…
Argo AI（Argo AI公司）；Carnegie Mellon University（卡内基梅隆大学）；Georgia Institute of Technology（佐治亚理工学院）
We present Argoverse -- two datasets designed to support autonomous vehicle machine learning tasks such as 3D tracking and motion forecasting. Argoverse was collected by a fleet of autonomous vehicles in Pittsburgh and Miami. The Argoverse 3D Tracking dataset includes 360 degree images from 7 cameras with overlapping fields of view, 3D point clouds from long range LiDAR, 6-DOF pose, and 3D track annotations. Notably, it is the only modern AV dataset that provides forward-facing stereo imagery. The Argoverse Motion Forecasting dataset includes more than 300,000 5-second tracked scenarios with a…
We introduce Argoverse 2 (AV2) - a collection of three datasets for perception and forecasting research in the self-driving domain. The annotated Sensor Dataset contains 1,000 sequences of multimodal data, encompassing high-resolution imagery from seven ring cameras, and two stereo cameras in addition to lidar point clouds, and 6-DOF map-aligned pose. Sequences contain 3D cuboid annotations for 26 object categories, all of which are sufficiently-sampled to support training and evaluation of 3D perception models. The Lidar Dataset contains 20,000 sequences of unlabeled lidar point clouds and ma…
IBM Research（IBM研究院）；IBM T.J. Watson Research Center（IBM托马斯·J·沃森研究中心）
This paper describes a new algorithm for solar energy forecasting from a sequence of Cloud Optical Depth (COD) images. The algorithm is based on the following simple observation: the dynamics of clouds represented by COD images resembles the motion (transport) of a density in a fluid flow. This suggests that, to forecast the motion of COD images, it is sufficient to forecast the flow. The latter, in turn, can be accomplished by fitting a parametric model of the fluid flow to the COD images observed in the past. Namely, the learning phase of the algorithm is composed of the following steps: (i)…
Nanyang Technological University (NTU)（南洋理工大学）；Advanced Digital Sciences Center (ADSC)（高级数字科学中心）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
The analysis of clouds in the earth's atmosphere is important for a variety of applications, viz. weather reporting, climate forecasting, and solar energy generation. In this paper, we focus our attention on the impact of cloud on the total solar irradiance reaching the earth's surface. We use weather station to record the total solar irradiance. Moreover, we employ collocated ground-based sky camera to automatically compute the instantaneous cloud coverage. We analyze the relationship between measured solar irradiance and computed cloud coverage value, and conclude that higher cloud coverage …
Self-supervised learning (SSL) is an emerging technique that has been successfully employed to train convolutional neural networks (CNNs) and graph neural networks (GNNs) for more transferable, generalizable, and robust representation learning. However its potential in motion forecasting for autonomous driving has rarely been explored. In this study, we report the first systematic exploration and assessment of incorporating self-supervision into motion forecasting. We first propose to investigate four novel self-supervised learning tasks for motion forecasting with theoretical rationale and qu…
Improving irradiance forecasting is critical to further increase the share of solar in the energy mix. On a short time scale, fish-eye cameras on the ground are used to capture cloud displacements causing the local variability of the electricity production. As most of the solar radiation comes directly from the Sun, current forecasting approaches use its position in the image as a reference to interpret the cloud cover dynamics. However, existing Sun tracking methods rely on external data and a calibration of the camera, which requires access to the device. To address these limitations, this s…
TREND: Unsupervised 3D Representation Learning via Temporal Forecasting for LiDAR Perception
The University of Hong Kong（香港大学）；Yale University（耶鲁大学）；Shanghai AI Laboratory（上海人工智能实验室）；HKU Shanghai Intelligent Computing Research Center（香港大学上海智能计算研究中心）
M3S-Net: Multimodal Feature Fusion Network Based on Multi-scale Data for Ultra-short-term PV Power Forecasting
School of Artificial Intelligence, Hebei University of Technology（人工智能学院，河北工业大学）；University of Southern Queensland（南方昆士兰大学）；School of Information Engineering, Tianjin University of Commerce（信息工程学院，天津商业大学）；Hebei Province Key Laboratory of Big Data Calculation, Hebei University of Technology（大数据计算河北重点实验室，河北工业大学）；General AI Lab, School of Engineering, Westlake University（通用人工智能实验室，西湖大学）；School of Business and Law, Edith Cowan University（商学院和法学院，埃迪斯科文大学）
University of Cambridge（剑桥大学）；ENGIE Lab CRIGEN（昂吉集团CRIGEN实验室）
A number of industrial applications, such as smart grids, power plant operation, hybrid system management or energy trading, could benefit from improved short-term solar forecasting, addressing the intermittent energy production from solar panels. However, current approaches to modelling the cloud cover dynamics from sky images still lack precision regarding the spatial configuration of clouds, their temporal dynamics and physical interactions with solar radiation. Benefiting from a growing number of large datasets, data driven methods are being developed to address these limitations with prom…
Deep Learning Super Resolution for Satellite Cloud Mask Downscaling
BEYOND EO Centre（BEYOND EO中心）；IAASARS, National Observatory of Athens（雅典国家天文台IAASARS研究所）；National and Kapodistrian University of Athens（雅典国立及卡波迪斯特里亚大学）
UniTS: Unified Spatio-Temporal Generative Model for Remote Sensing
Jockey Club STEM Laboratory of Quantitative Remote Sensing（裘英俊STEM实验室（定量遥感））；Department of Geography, the University of Hong Kong（香港大学地理系）；School of Information and Electronics, Beijing Institute of Technology（北京理工大学信息电子学院）；Beijing Key Laboratory of Fractional Signals and Systems（北京分数信号与系统重点实验室）；Department of Electrical and Computer Engineering, University of Delaware（德雷塞尔大学电气与计算机工程系）
Segregation and Context Aggregation Network for Real-time Cloud Segmentation
Carnegie Mellon University（卡内基梅隆大学）；University of Nottingham（诺丁汉大学）；The University of Hong Kong（香港大学）；Cornell University（康奈尔大学）；University College Dublin（都柏林大学）
Despite the advances in the field of solar energy, improvements of solar forecasting techniques, addressing the intermittent electricity production, remain essential for securing its future integration into a wider energy supply. A promising approach to anticipate irradiance changes consists of modeling the cloud cover dynamics from ground taken or satellite images. This work presents preliminary results on the application of deep Convolutional Neural Networks for 2 to 20 min irradiance forecasting using hemispherical sky images and exogenous variables. We evaluate the models on a set of irrad…
