Large-area crop classification using multi-spectral imagery is a widely studied problem for several decades and is generally addressed using classical Random Forest classifier. Recently, deep convolutional neural networks (DCNN) have been proposed. However, these methods only achieved results comparable with Random Forest. In this work, we present a novel CNN based architecture for large-area crop classification. Our methodology combines both spatio-temporal analysis via 3D CNN as well as temporal analysis via 1D CNN. We evaluated the efficacy of our approach on Yolo and Imperial county benchmark datasets. Our combined strategy outperforms both classical as well as recent DCNN based methods in terms of classification accuracy by 2% while maintaining a minimum number of parameters and the lowest inference time.
We propose Spatio-temporal Crop Aggregation for video representation LEarning (SCALE), a novel method that enjoys high scalability at both training and inference time. Our model builds long-range video features by learning from sets of video clip-level features extracted with a pre-trained backbone. To train the model, we propose a self-supervised objective consisting of masked clip feature prediction. We apply sparsity to both the input, by extracting a random set of video clips, and to the loss function, by only reconstructing the sparse inputs. Moreover, we use dimensionality reduction by w…
Georgia Institute of Technology（佐治亚理工学院）；University of Georgia（佐治亚大学）
Autonomous crop monitoring at high spatial and temporal resolution is a critical problem in precision agriculture. While Structure from Motion and Multi-View Stereo algorithms can finely reconstruct the 3D structure of a field with low-cost image sensors, these algorithms fail to capture the dynamic nature of continuously growing crops. In this paper we propose a 4D reconstruction approach to crop monitoring, which employs a spatio-temporal model of dynamic scenes that is useful for precision agriculture applications. Additionally, we provide a robust data association algorithm to address the …
Land use classification of low resolution spatial imagery is one of the most extensively researched fields in remote sensing. Despite significant advancements in satellite technology, high resolution imagery lacks global coverage and can be prohibitively expensive to procure for extended time periods. Accurately classifying land use change without high resolution imagery offers the potential to monitor vital aspects of global development agenda including climate smart agriculture, drought resistant crops, and sustainable land management. Utilizing a combination of capsule layers and long-short…
A multi-temporal multi-spectral attention-augmented deep convolution neural network with contrastive learning for crop yield prediction
Department of Computer Science and Engineering, Indian Institute of Technology (IIT) Indore, Indore 453552, India（计算机科学与工程系，印度理工学院（IIT）印多尔，印多尔453552，印度）
Indraprastha Institute of Information Technology, Delhi（德里英迪拉普拉斯塔信息技术学院）；M S Swaminathan Research Foundation（M·S·斯瓦米纳坦研究基金会）
The integration of the modern Machine Learning (ML) models into remote sensing and agriculture has expanded the scope of the application of satellite images in the agriculture domain. In this paper, we present how the accuracy of crop type identification improves as we move from medium-spatiotemporal-resolution (MSTR) to high-spatiotemporal-resolution (HSTR) satellite images. We further demonstrate that high spectral resolution in satellite imagery can improve prediction performance for low spatial and temporal resolutions (LSTR) images. The F1-score is increased by 7% when using multispectral…
Spatio-temporal action detection is an important and challenging problem in video understanding. However, the application of the existing large-scale spatio-temporal action datasets in specific fields is limited, and there is currently no public tool for making spatio-temporal action datasets, it takes a lot of time and effort for researchers to customize the spatio-temporal action datasets, so we propose a multi-Person video dataset Annotation Method of spatio-temporally actions.First, we use ffmpeg to crop the videos and frame the videos; then use yolov5 to detect human in the video frame, a…
We present a robust method to correct for motion and deformations for in-utero volumetric MRI time series. Spatio-temporal analysis of dynamic MRI requires robust alignment across time in the presence of substantial and unpredictable motion. We make a Markov assumption on the nature of deformations to take advantage of the temporal structure in the image data. Forward message passing in the corresponding hidden Markov model (HMM) yields an estimation algorithm that only has to account for relatively small motion between consecutive frames. We demonstrate the utility of the temporal model by sh…
Tree crop mapping of South America reveals links to deforestation and conservation
Google DeepMind（谷歌DeepMind）；EcoVision Lab, DM3L, University of Zurich（EcoVision实验室、DM3L、苏黎世大学）；University of Zurich（苏黎世大学）；University of Copenhagen（哥本哈根大学）；Google Research（谷歌研究）；ETH Zürich（苏黎世联邦理工学院）
Realizing Immersive Volumetric Video: A Multimodal Framework for 6-DoF VR Engagement
Tsinghua University（清华大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Migu Beijing Research Institute（咪咕北京研究院）；School of Architecture, Tsinghua University（清华大学建筑学院）；Department of Automation, Tsinghua University（清华大学自动化系）；BNRist, Tsinghua University（清华大学北京信息科学与技术国家研究中心）
AgriFM: A Multi-source Temporal Remote Sensing Foundation Model for Agriculture Mapping
Jockey Club STEM Lab of Quantitative Remote Sensing, Department of Geography, The University of Hong Kong, Hong Kong, China（香港大学地理系金钟STEM实验室）；Department of Aerospace Intelligent Science and Technology, School of Astronautics, Beihang University, Beijing, China（北航航天智能科学与技术学院）；School of Remote Sensing and Information Engineering, Wuhan University, China（武汉大学遥感与信息工程学院）
Robust forecasting of the future anatomical changes inflicted by an ongoing disease is an extremely challenging task that is out of grasp even for experienced healthcare professionals. Such a capability, however, is of great importance since it can improve patient management by providing information on the speed of disease progression already at the admission stage, or it can enrich the clinical trials with fast progressors and avoid the need for control arms by the means of digital twins. In this work, we develop a deep learning method that models the evolution of age-related disease by proce…
Boosting Crop Classification by Hierarchically Fusing Satellite, Rotational, and Contextual Data
Centro Nacional de Inteligencia Artificial (CENIA)（国家人工智能中心）；European Commission, Joint Research Centre (JRC)（欧盟委员会联合研究中心）；Technical University of Munich（慕尼黑工业大学）
Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡大学）；Massachusetts Institute of Technology（麻省理工学院）
Optical coherence tomography (OCT) enables high-resolution and non-invasive 3D imaging of the human retina but is inherently impaired by speckle noise. This paper introduces a spatio-temporal denoising algorithm for OCT data on a B-scan level using a novel quantile sparse image (QuaSI) prior. To remove speckle noise while preserving image structures of diagnostic relevance, we implement our QuaSI prior via median filter regularization coupled with a Huber data fidelity model in a variational approach. For efficient energy minimization, we develop an alternating direction method of multipliers …
Estimating the forces acting between instruments and tissue is a challenging problem for robot-assisted minimally-invasive surgery. Recently, numerous vision-based methods have been proposed to replace electro-mechanical approaches. Moreover, optical coherence tomography (OCT) and deep learning have been used for estimating forces based on deformation observed in volumetric image data. The method demonstrated the advantage of deep learning with 3D volumetric data over 2D depth images for force estimation. In this work, we extend the problem of deep learning-based force estimation to 4D spatio-…
Spatio-temporal feature learning is of central importance for action recognition in videos. Existing deep neural network models either learn spatial and temporal features independently (C2D) or jointly with unconstrained parameters (C3D). In this paper, we propose a novel neural operation which encodes spatio-temporal features collaboratively by imposing a weight-sharing constraint on the learnable parameters. In particular, we perform 2D convolution along three orthogonal views of volumetric video data,which learns spatial appearance and temporal motion cues respectively. By sharing the convo…
We introduce HUMAN4D, a large and multimodal 4D dataset that contains a variety of human activities simultaneously captured by a professional marker-based MoCap, a volumetric capture and an audio recording system. By capturing 2 female and $2$ male professional actors performing various full-body movements and expressions, HUMAN4D provides a diverse set of motions and poses encountered as part of single- and multi-person daily, physical and social activities (jumping, dancing, etc.), along with multi-RGBD (mRGBD), volumetric and audio data. Despite the existence of multi-view color datasets ca…
Department of Mathematics, University of Innsbruck（因斯布鲁克大学数学系）
Convolutional neural networks are state-of-the-art for various segmentation tasks. While for 2D images these networks are also computationally efficient, 3D convolutions have huge storage requirements and therefore, end-to-end training is limited by GPU memory and data size. To overcome this issue, we introduce a network structure for volumetric data without 3D convolution layers. The main idea is to include projections from different directions to transform the volumetric data to a sequence of images, where each image contains information of the full data. We then apply 2D convolutions to the…
Tracking and localizing objects is a central problem in computer-assisted surgery. Optical coherence tomography (OCT) can be employed as an optical tracking system, due to its high spatial and temporal resolution. Recently, 3D convolutional neural networks (CNNs) have shown promising performance for pose estimation of a marker object using single volumetric OCT images. While this approach relied on spatial information only, OCT allows for a temporal stream of OCT image volumes capturing the motion of an object at high volumes rates. In this work, we systematically extend 3D CNNs to 4D spatio-t…
We present an attention-based model that reasons on human body shape and motion dynamics to identify individuals in the absence of RGB information, hence in the dark. Our approach leverages unique 4D spatio-temporal signatures to address the identification problem across days. Formulated as a reinforcement learning task, our model is based on a combination of convolutional and recurrent neural networks with the goal of identifying small, discriminative regions indicative of human identity. We demonstrate that our model produces state-of-the-art results on several published datasets given only …
Self-supervised skeleton-based action recognition with contrastive learning has attracted much attention. Recent literature shows that data augmentation and large sets of contrastive pairs are crucial in learning such representations. In this paper, we found that directly extending contrastive pairs based on normal augmentations brings limited returns in terms of performance, because the contribution of contrastive pairs from the normal data augmentation to the loss get smaller as training progresses. Therefore, we delve into hard contrastive pairs for contrastive learning. Motivated by the su…
Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS（弗劳恩霍夫智能分析与信息系统研究所（IAIS））；University of Bonn（波恩大学）；Institute for Sugar Beet Research (IfZ)（甜菜研究所（IfZ））；Technical University of Munich（慕尼黑工业大学）
UAV-based image retrieval in modern agriculture enables gathering large amounts of spatially referenced crop image data. In large-scale experiments, however, UAV images suffer from containing a multitudinous amount of crops in a complex canopy architecture. Especially for the observation of temporal effects, this complicates the recognition of individual plants over several images and the extraction of relevant information tremendously. In this work, we present a hands-on workflow for the automatized temporal and spatial identification and individualization of crop images from UAVs abbreviated…
Historical maps provide useful spatio-temporal information on the Earth's surface before modern earth observation techniques came into being. To extract information from maps, neural networks, which gain wide popularity in recent years, have replaced hand-crafted map processing methods and tedious manual labor. However, aleatoric uncertainty, known as data-dependent uncertainty, inherent in the drawing/scanning/fading defects of the original map sheets and inadequate contexts when cropping maps into small tiles considering the memory limits of the training process, challenges the model to make…
PAII Inc.（PAII公司）；Johns Hopkins University（约翰斯·霍普金斯大学）；Nvidia Corporation（英伟达公司）；National Institutes of Health Clinical Center（美国国立卫生研究院临床中心）；Tencent Holdings Limited（腾讯控股有限公司）
Prognostic tumor growth modeling via volumetric medical imaging observations can potentially lead to better outcomes of tumor treatment and surgical planning. Recent advances of convolutional networks have demonstrated higher accuracy than traditional mathematical models in predicting future tumor volumes. This indicates that deep learning-based techniques may have great potentials on addressing such problem. However, current 2D patch-based modeling approaches cannot make full use of the spatio-temporal imaging context of the tumor's longitudinal 4D (3D + time) data. Moreover, they are incapab…
Tel Aviv University（特拉维夫大学）；Facebook AI Research（脸书人工智能研究院）
Recently developed methods for rapid continuous volumetric two-photon microscopy facilitate the observation of neuronal activity in hundreds of individual neurons and changes in blood flow in adjacent blood vessels across a large volume of living brain at unprecedented spatio-temporal resolution. However, the high imaging rate necessitates fully automated image analysis, whereas tissue turbidity and photo-toxicity limitations lead to extremely sparse and noisy imagery. In this work, we extend a recently proposed deep learning volumetric blood vessel segmentation network, such that it supports …
University of Minnesota（明尼苏达大学）；University of Pittsburgh（匹兹堡大学）
The availability of massive earth observing satellite data provide huge opportunities for land use and land cover mapping. However, such mapping effort is challenging due to the existence of various land cover classes, noisy data, and the lack of proper labels. Also, each land cover class typically has its own unique temporal pattern and can be identified only during certain periods. In this article, we introduce a novel architecture that incorporates the UNet structure with Bidirectional LSTM and Attention mechanism to jointly exploit the spatial and temporal nature of satellite data and to b…
Interlaced dynamic XCT reconstruction with spatio-temporal implicit neural representations
Nuclear Science and Engineering, Massachusetts Institute of Technology（麻省理工学院核科学与工程系）；École Polytechnique de Bruxelles, Université Libre de Bruxelles（布鲁塞尔联邦理工学院）；Electrical Engineering and Computer Science, Massachusetts Institute of Technology（麻省理工学院电气工程与计算机科学系）
Digital Typhoon: Long-term Satellite Image Dataset for the Spatio-Temporal Modeling of Tropical Cyclones
National Institute of Informatics（国立情报学研究所）；Yokohama National University（横滨国立大学）；University of Southern California（南加州大学）；Grenoble-INP, Ensimag（格勒诺布尔综合理工学院，恩西马格）；Université Clermont Auvergne, ISIMA（克莱蒙奥弗涅大学，ISIMA）；Google DeepMind（谷歌DeepMind）
Concordia University（康考迪亚大学）；Concordia Institute for Information Systems Engineering（康考迪亚信息系统工程研究所）；Sunnybrook Health Sciences Centre（桑尼布鲁克健康科学中心）；University of Toronto（多伦多大学）
The paper proposes a novel hybrid discovery Radiomics framework that simultaneously integrates temporal and spatial features extracted from non-thin chest Computed Tomography (CT) slices to predict Lung Adenocarcinoma (LUAC) malignancy with minimum expert involvement. Lung cancer is the leading cause of mortality from cancer worldwide and has various histologic types, among which LUAC has recently been the most prevalent. LUACs are classified as pre-invasive, minimally invasive, and invasive adenocarcinomas. Timely and accurate knowledge of the lung nodules malignancy leads to a proper treatme…
