Predicting Crop Yield With Machine Learning: An Extensive Analysis Of Input Modalities And Models On a Field and sub-field Level
University of Kaiserslautern-Landau (RPTU)（凯泽斯劳滕-兰道大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Vision Impulse GmbH（Vision Impulse 有限责任公司）
Deepak Pathak, Miro Miranda, Francisco Mena, Cristhian Sanchez, Patrick Helber, Benjamin Bischke, Peter Habelitz, Hiba Najjar, Jayanth Siddamsetty, Diego Arenas… 展开作者
Deepak Pathak, Miro Miranda, Francisco Mena, Cristhian Sanchez, Patrick Helber, Benjamin Bischke, Peter Habelitz, Hiba Najjar, Jayanth Siddamsetty, Diego Arenas, Michaela Vollmer, Marcela Charfuelan, Marlon Nuske, Andreas Dengel
We introduce a simple yet effective early fusion method for crop yield prediction that handles multiple input modalities with different temporal and spatial resolutions. We use high-resolution crop yield maps as ground truth data to train crop and machine learning model agnostic methods at the sub-field level. We use Sentinel-2 satellite imagery as the primary modality for input data with other complementary modalities, including weather, soil, and DEM data. The proposed method uses input modalities available with global coverage, making the framework globally scalable. We explicitly highlight the importance of input modalities for crop yield prediction and emphasize that the best-performing combination of input modalities depends on region, crop, and chosen model.
Fine-grained Hierarchical Crop Type Classification from Integrated Hyperspectral EnMAP Data and Multispectral Sentinel-2 Time Series: A Large-scale Dataset and Dual-stream Transformer Method
Jockey Club STEM Lab of Quantitative Remote Sensing, Department of Geography, The University of Hong Kong, Hong Kong, China（香港大学地理系定量遥感实验室）；Department of Aerospace Intelligent Science and Technology, School of Astronautics, Beihang University, Beijing, China（北航航天智能科学与技术系）
Soil moisture is critical component of crop health and monitoring it can enable further actions for increasing yield or preventing catastrophic die off. As climate change increases the likelihood of extreme weather events and reduces the predictability of weather, and non-optimal soil moistures for crops may become more likely. In this work, we a series of LSTM architectures to analyze measurements of soil moisture and vegetation indiced derived from satellite imagery. The system learns to predict the future values of these measurements. These spatially sparse values and indices are used as in…
Indraprastha Institute of Information Technology, Delhi（德里英德拉普拉斯塔信息技术学院）；M S Swaminathan Research Foundation（M S 斯瓦米纳坦研究基金会）
The availability of well-curated datasets has driven the success of Machine Learning (ML) models. Despite the increased access to earth observation data for agriculture, there is a scarcity of curated, labelled datasets, which limits the potential of its use in training ML models for remote sensing (RS) in agriculture. To this end, we introduce a first-of-its-kind dataset, SICKLE, having time-series images at different spatial resolutions from 3 different satellites, annotated with multiple key cropping parameters for paddy cultivation for the Cauvery Delta region in Tamil Nadu, India. The dat…
University of Delaware（特拉华大学）；University of Louisiana at Lafayette（路易斯安那大学拉斐特分校）；University of Connecticut（康涅狄格大学）；University of Oklahoma（俄克拉荷马大学）；Mississippi State University（密西西比州立大学）；Louisiana State University（路易斯安那州立大学）；LSU AgCenter（路易斯安那州立大学农业中心）；Tulane University（杜兰大学）
Crossmodal learning for Crop Canopy Trait Estimation
Department of Computer Engineering, Iowa State University, Ames, IA, USA（计算机工程系，爱荷华州立大学）；Department of Mechanical Engineering, Iowa State University, Ames, IA, USA（机械工程系，爱荷华州立大学）
St. Pölten University of Applied Sciences（圣珀尔滕应用科学大学）；Kufstein University of Applied Sciences（库夫施泰因应用科学大学）
The assessment and valuation of real estate requires large datasets with real estate information. Unfortunately, real estate databases are usually sparse in practice, i.e., not for each property every important attribute is available. In this paper, we study the potential of predicting high-level real estate attributes from visual data, specifically from two visual modalities, namely indoor (interior) and outdoor (facade) photos. We design three models using different multimodal fusion strategies and evaluate them for three different use cases. Thereby, a particular challenge is to handle miss…
Boosting Crop Classification by Hierarchically Fusing Satellite, Rotational, and Contextual Data
Centro Nacional de Inteligencia Artificial (CENIA)（国家人工智能中心）；European Commission, Joint Research Centre (JRC)（欧盟委员会联合研究中心）；Technical University of Munich（慕尼黑工业大学）
To study neurodegenerative diseases, longitudinal studies are carried on volunteer patients. During a time span of several months to several years, they go through regular medical visits to acquire data from different modalities, such as biological samples, cognitive tests, structural and functional imaging. These variables are heterogeneous but they all depend on the patient's health condition, meaning that there are possibly unknown relationships between all modalities. Some information may be specific to some modalities, others may be complementary, and others may be redundant. Some data ma…
Goldman School of Public Policy, UC Berkeley（加州大学伯克利分校高盛公共政策学院）；Institute for Computational and Mathematical Engineering, Stanford University（斯坦福大学计算与数学工程研究所）；Department of Earth System Science, Stanford University（斯坦福大学地球系统科学系）；European Commission Joint Research Centre（欧盟委员会联合研究中心）
Crop field boundaries aid in mapping crop types, predicting yields, and delivering field-scale analytics to farmers. Recent years have seen the successful application of deep learning to delineating field boundaries in industrial agricultural systems, but field boundary datasets remain missing in smallholder systems due to (1) small fields that require high resolution satellite imagery to delineate and (2) a lack of ground labels for model training and validation. In this work, we combine transfer learning and weak supervision to overcome these challenges, and we demonstrate the methods' succe…
Galileo: Learning Global & Local Features of Many Remote Sensing Modalities
Mila -- Quebec AI Institute（魁北克人工智能研究所）；McGill University（麦吉尔大学）；Arizona State University（亚利桑那州立大学）；Carleton University（卡尔顿大学）；Allen Institute for AI (Ai2)（人工智能研究所）；University of British Columbia（不列颠哥伦比亚大学）；Vector Institute（向量研究所）
In the semantic segmentation of street scenes the reliability of the prediction and therefore uncertainty measures are of highest interest. We present a method that generates for each input image a hierarchy of nested crops around the image center and presents these, all re-scaled to the same size, to a neural network for semantic segmentation. The resulting softmax outputs are then post processed such that we can investigate mean and variance over all image crops as well as mean and variance of uncertainty heat maps obtained from pixel-wise uncertainty measures, like the entropy, applied to e…
National University of Singapore（新加坡国立大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Goethe University Frankfurt（法兰克福大学）
Although media content is increasingly produced, distributed, and consumed in multiple combinations of modalities, how individual modalities contribute to the perceived emotion of a media item remains poorly understood. In this paper we present MusicVideos (MuVi), a novel dataset for affective multimedia content analysis to study how the auditory and visual modalities contribute to the perceived emotion of media. The data were collected by presenting music videos to participants in three conditions: music, visual, and audiovisual. Participants annotated the music videos for valence and arousal…
A multi-temporal multi-spectral attention-augmented deep convolution neural network with contrastive learning for crop yield prediction
Department of Computer Science and Engineering, Indian Institute of Technology (IIT) Indore, Indore 453552, India（计算机科学与工程系，印度理工学院（IIT）印多尔，印多尔453552，印度）
Alibaba Group（阿里巴巴集团）；South China University of Technology（华南理工大学）
Predicting the emotional impact of videos using machine learning is a challenging task considering the varieties of modalities, the complicated temporal contex of the video as well as the time dependency of the emotional states. Feature extraction, multi-modal fusion and temporal context fusion are crucial stages for predicting valence and arousal values in the emotional impact, but have not been successfully exploited. In this paper, we propose a comprehensive framework with novel designs of modal structure and multi-modal fusion strategy. We select the most suitable modalities for valence an…
University of Washington（华盛顿大学）；Texas A&M University（德克萨斯A&M大学）
We present our preliminary work to determine if patient's vocal acoustic, linguistic, and facial patterns could predict clinical ratings of depression severity, namely Patient Health Questionnaire depression scale (PHQ-8). We proposed a multi modal fusion model that combines three different modalities: audio, video , and text features. By training over AVEC 2017 data set, our proposed model outperforms each single modality prediction model, and surpasses the data set baseline with ice margin.
Multimodal Doctor-in-the-Loop: A Clinically-Guided Explainable Framework for Predicting Pathological Response in Non-Small Cell Lung Cancer
Research Unit of Computer Systems and Bioinformatics, Department of Engineering（计算机系统与生物信息学研究单位，工程系）；Research Unit of Radiation Oncology, Department of Medicine and Surgery（放射肿瘤学研究单位，医学与外科系）；Research Unit of Anatomical Pathology, Department of Medicine and Surgery（病理学研究单位，医学与外科系）
Predicting future sensory states is crucial for learning agents such as robots, drones, and autonomous vehicles. In this paper, we couple multiple sensory modalities with exploratory actions and propose a predictive neural network architecture to address this problem. Most existing approaches rely on large, manually annotated datasets, or only use visual data as a single modality. In contrast, the unsupervised method presented here uses multi-modal perceptions for predicting future visual frames. As a result, the proposed model is more comprehensive and can better capture the spatio-temporal d…
Genomics data such as RNA gene expression, methylation and micro RNA expression are valuable sources of information for various clinical predictive tasks. For example, predicting survival outcomes, cancer histology type and other patients' related information is possible using not only clinical data but molecular data as well. Moreover, using these data sources together, for example in multitask learning, can boost the performance. However, in practice, there are many missing data points which leads to significantly lower patient numbers when analysing full cases, which in our setting refers t…
Fusion of Diffusion Weighted MRI and Clinical Data for Predicting Functional Outcome after Acute Ischemic Stroke with Deep Contrastive Learning
Queens College of the City University of NY（纽约市立大学皇后学院）；National Chung Cheng University（台湾中正大学）；Ditmanson Medical Foundation Chia-Yi Christian Hospital（戴德森医疗财团法人嘉义基督教医院）
Predicting the behavior of road users, particularly pedestrians, is vital for safe motion planning in the context of autonomous driving systems. Traditionally, pedestrian behavior prediction has been realized in terms of forecasting future trajectories. However, recent evidence suggests that predicting higher-level actions, such as crossing the road, can help improve trajectory forecasting and planning tasks accordingly. There are a number of existing datasets that cater to the development of pedestrian action prediction algorithms, however, they lack certain characteristics, such as bird's ey…
Arbitrary Data as Images: Fusion of Patient Data Across Modalities and Irregular Intervals with Vision Transformers
Heidelberg University Hospital（海德堡大学医院）；Informatics for Life Institute（生命信息学研究所）；German Centre for Cardiovascular Research（德国心血管研究中心）；University Medical Center of the Johannes Gutenberg University Mainz（美因茨约翰内斯·古滕贝格大学医学中心）；University Medical Center Mainz（美因茨大学医学中心）；Institute of Molecular Biology (IMB)（分子生物学研究所）
MIT（麻省理工学院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；Tsinghua University（清华大学）
Humans integrate multiple sensory modalities (e.g. visual and audio) to build a causal understanding of the physical world. In this work, we propose a novel type of intrinsic motivation for Reinforcement Learning (RL) that encourages the agent to understand the causal effect of its actions through auditory event prediction. First, we allow the agent to collect a small amount of acoustic data and use K-means to discover underlying auditory event clusters. We then train a neural network to predict the auditory events and use the prediction errors as intrinsic rewards to guide RL exploration. Exp…
Kansas State University（堪萨斯州立大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
With millions of images that are shared online on social networking sites, effective methods for image privacy prediction are highly needed. In this paper, we propose an approach for fusing object, scene context, and image tags modalities derived from convolutional neural networks for accurately predicting the privacy of images shared online. Specifically, our approach identifies the set of most competent modalities on the fly, according to each new target image whose privacy has to be predicted. The approach considers three stages to predict the privacy of a target image, wherein we first ide…
IMT Lille Douai（IMT里尔杜埃）；CRIStAL UMR CNRS 9189（法国国家科学研究中心信息、自动化与信号处理实验室）；University of Mons（蒙斯大学）
Visual attention estimation is an active field of research at the crossroads of different disciplines: computer vision, artificial intelligence and medicine. One of the most common approaches to estimate a saliency map representing attention is based on the observed images. In this paper, we show that visual attention can be retrieved from EEG acquisition. The results are comparable to traditional predictions from observed images, which is of great interest. For this purpose, a set of signals has been recorded and different models have been developed to study the relationship between visual at…
Generating Multimodal Driving Scenes via Next-Scene Prediction
School of Software Engineering, XJTU（西安交通大学软件学院）；Horizon Robotics（地平线机器人）；School of Computer and Communication Sciences, EPFL（洛桑联邦理工学院计算机与通信科学学院）；University of Chinese Academy of Sciences（中国科学院大学）
University of Minnesota（明尼苏达大学）；University of California-Davis（加州大学戴维斯分校）
Crop production needs to increase in a sustainable manner to meet the growing global demand for food. To identify crop varieties with high yield potential, plant scientists and breeders evaluate the performance of hundreds of lines in multiple locations over several years. To facilitate the process of selecting advanced varieties, an automated framework was developed in this study. A hyperspectral camera was mounted on an unmanned aerial vehicle to collect aerial imagery with high spatial and spectral resolution. Aerial images were captured in two consecutive growing seasons from three experim…
University of Catania（卡塔尼亚大学）；Philips Research（飞利浦研究院）
Egocentric videos can bring a lot of information about how humans perceive the world and interact with the environment, which can be beneficial for the analysis of human behaviour. The research in egocentric video analysis is developing rapidly thanks to the increasing availability of wearable devices and the opportunities offered by new large-scale egocentric datasets. As computer vision techniques continue to develop at an increasing pace, the tasks related to the prediction of future are starting to evolve from the need of understanding the present. Predicting future human activities, traje…
Textual data such as tags, sentence descriptions are combined with visual cues to reduce the semantic gap for image retrieval applications in today's Multimodal Image Retrieval (MIR) systems. However, all tags are treated as equally important in these systems, which may result in misalignment between visual and textual modalities during MIR training. This will further lead to degenerated retrieval performance at query time. To address this issue, we investigate the problem of tag importance prediction, where the goal is to automatically predict the tag importance and use it in image retrieval.…
Learning from Limited and Incomplete Data: A Multimodal Framework for Predicting Pathological Response in NSCLC
Department of Medicine and Surgery（医学与外科系）；Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering（诊断与介入系，放射物理，生物医学工程）；Umeå University, Umeå, Sweden（乌梅大学，瑞典）
Singapore University of Technology and Design（新加坡科技设计大学）；Goethe University Frankfurt（法兰克福歌德大学）
In this work, we propose different variants of the self-attention based network for emotion prediction from movies, which we call AttendAffectNet. We take both audio and video into account and incorporate the relation among multiple modalities by applying self-attention mechanism in a novel manner into the extracted features for emotion prediction. We compare it to the typically temporal integration of the self-attention based model, which in our case, allows to capture the relation of temporal representations of the movie while considering the sequential dependencies of emotion responses. We …
École de Technologie Supérieure, University of Québec（魁北克大学高等技术学院）
This paper presents a novel deep neural network (DNN) for multimodal fusion of audio, video and text modalities for emotion recognition. The proposed DNN architecture has independent and shared layers which aim to learn the representation for each modality, as well as the best combined representation to achieve the best prediction. Experimental results on the AVEC Sentiment Analysis in the Wild dataset indicate that the proposed DNN can achieve a higher level of Concordance Correlation Coefficient (CCC) than other state-of-the-art systems that perform early fusion of modalities at feature-leve…
