This paper proposes a strategy for visual prediction in the context of autonomous driving. Humans, when not distracted or drunk, are still the best drivers you can currently find. For this reason we take inspiration from two theoretical ideas about the human mind and its neural organization. The first idea concerns how the brain uses a hierarchical structure of neuron ensembles to extract abstract concepts from visual experience and code them into compact representations. The second idea suggests that these neural perceptual representations are not neutral but functional to the prediction of the future state of affairs in the environment. Similarly, the prediction mechanism is not neutral but oriented to the current planning of a future action. We identify within the deep learning framework two artificial counterparts of the aforementioned neurocognitive theories. We find a correspondence between the first theoretical idea and the architecture of convolutional autoencoders, while we translate the second theory into a training procedure that learns compact representations which are not neutral but oriented to driving tasks, from two distinct perspectives. From a static perspective, we force groups of neural units in the compact representations to distinctly represent specific concepts crucial to the driving task. From a dynamic perspective, we encourage the compact representations to be predictive of how the current road scenario will change in the future. We successfully learn compact representations that use as few as 16 neural units for each of the two basic driving concepts we consider: car and lane. We prove the efficiency of our proposed perceptual representations on the SYNTHIA dataset. Our source code is available at https://github.com/3lis/rnn_vae
Physics informed Transformer-VAE for biophysical parameter estimation: PROSAIL model inversion in Sentinel-2 imagery
African Institute for Mathematical Sciences (AIMS), Accra, Ghana（非洲数学科学研究所（AIMS））；African Institute for Mathematical Sciences (AIMS), Cape Town, South Africa（非洲数学科学研究所（AIMS））
NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagery
University of Minnesota（明尼苏达大学）；Alljoined；Princeton Neuroscience Institute（普林斯顿神经科学研究所）；Stability AI/Medical AI Research Center (MedARC)（Stability AI/医疗AI研究中心（MedARC））
The EmbodiedQA is a task of training an embodied agent by intelligently navigating in a simulated environment and gathering visual information to answer questions. Existing approaches fail to explicitly model the mental imagery function of the agent, while the mental imagery is crucial to embodied cognition, and has a close relation to many high-level meta-skills such as generalization and interpretation. In this paper, we propose a novel Mental Imagery eNhanceD (MIND) module for the embodied agent, as well as a relevant deep reinforcement framework for training. The MIND module can not only m…
MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery
University of Minnesota（明尼苏达大学）；Medical AI Research Center (MedARC)（医学人工智能研究中心 (MedARC)）；University of Sydney（悉尼大学）；Stanford University（斯坦福大学）；Alljoined；Sophont；Princeton Neuroscience Institute（普林斯顿神经科学研究所）
Hypnopaedia-Aware Machine Unlearning via Psychometrics of Artificial Mental Imagery
National Institute of Informatics（国家信息研究所）；Feng Chia University（逢甲大学）；Lancaster University（兰卡斯特大学）；University of Melbourne（墨尔本大学）；Graduate University for Advanced Studies（高级研究大学）；University of Tokyo（东京大学）
Mind-to-Image: Projecting Visual Mental Imagination of the Brain from fMRI
Obvious Research（奥布维研究机构）；GHU Paris Psychiatrie et Neurosciences（巴黎精神病学与神经科学大学医院集团）；Université Paris Cité（巴黎西岱大学）；Centre Imagerie du Nord（北方影像中心）；Raidium（雷迪姆公司）；FrontLab, Sorbonne Université（索邦大学前沿实验室）；Institut du Cerveau（大脑研究所）；Centre de NeuroImagerie de Recherche, Sorbonne Université（索邦大学神经影像研究中心）；Institut des Systèmes Intelligents et de la Robotique, Sorbonne Université（索邦大学智能系统与机器人研究所）
Learning image transformations is essential to the idea of mental simulation as a method of cognitive inference. We take a connectionist modeling approach, using planar neural networks to learn fundamental imagery transformations, like translation, rotation, and scaling, from perceptual experiences in the form of image sequences. We investigate how variations in network topology, training data, and image shape, among other factors, affect the efficiency and effectiveness of learning visual imagery transformations, including effectiveness of transfer to operating on new types of data.
The problem of supervised classification of the satellite image is considered to be the task of grouping pixels into a number of homogeneous regions in space intensity. This paper proposes a novel approach that combines a radial basic function clustering network with a growing neural gas include utility factor classifier to yield improved solutions, obtained with previous networks. The double objective technique is first used to the development of a method to perform the satellite images classification, and finally, the implementation to address the issue of the number of nodes in the hidden l…
Manchester Metropolitan University（曼彻斯特城市大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；Brunel University（布鲁内尔大学）
Spectral-spatial based deep learning models have recently proven to be effective in hyperspectral image (HSI) classification for various earth monitoring applications such as land cover classification and agricultural monitoring. However, due to the nature of "black-box" model representation, how to explain and interpret the learning process and the model decision, especially for vegetation classification, remains an open challenge. This study proposes a novel interpretable deep learning model -- a biologically interpretable two-stage deep neural network (BIT-DNN), by incorporating the prior-k…
Northumbria University at Newcastle（纽卡斯尔诺森比亚大学）；College of Computer and Information Sciences, King Saud University（沙特国王大学计算机与信息科学学院）
Colorectal and prostate cancers are the most common types of cancer in men worldwide. To diagnose colorectal and prostate cancer, a pathologist performs a histological analysis on needle biopsy samples. This manual process is time-consuming and error-prone, resulting in high intra and interobserver variability, which affects diagnosis reliability. This study aims to develop an automatic computerized system for diagnosing colorectal and prostate tumors by using images of biopsy samples to reduce time and diagnosis error rates associated with human analysis. We propose a CNN model for classifyin…
Technical University of Munich（慕尼黑工业大学）；ETH Zurich（苏黎世联邦理工学院）；Massachusetts Institute of Technology（麻省理工学院）
Forest carbon offsets are increasingly popular and can play a significant role in financing climate mitigation, forest conservation, and reforestation. Measuring how much carbon is stored in forests is, however, still largely done via expensive, time-consuming, and sometimes unaccountable field measurements. To overcome these limitations, many verification bodies are leveraging machine learning (ML) algorithms to estimate forest carbon from satellite or aerial imagery. Aerial imagery allows for tree species or family classification, which improves the satellite imagery-based forest type classi…
The Amazon rainforests have been suffering widespread damage, both via natural and artificial means. Every minute, it is estimated that the world loses forest cover the size of 48 football fields. Deforestation in the Amazon rainforest has led to drastically reduced biodiversity, loss of habitat, climate change, and other biological losses. In this respect, it has become essential to track how the nature of these forests change over time. Image classification using deep learning can help speed up this process by removing the manual task of classifying each image. Here, it is shown how convolut…
The Hong Kong Polytechnic University（香港理工大学）；Bavarian Forest National Park（巴伐利亚森林国家公园）；Albert-Ludwigs-Universität Freiburg（弗赖堡大学）
The stability and ability of an ecosystem to withstand climate change is directly linked to its biodiversity. Dead trees are a key indicator of overall forest health, housing one-third of forest ecosystem biodiversity, and constitute 8%of the global carbon stocks. They are decomposed by several natural factors, e.g. climate, insects and fungi. Accurate detection and modeling of dead wood mass is paramount to understanding forest ecology, the carbon cycle and decomposers. We present a novel method to construct precise shape contours of dead trees from aerial photographs by combining established…
Beihang University（北京航空航天大学）；The University of Sheffield（谢菲尔德大学）
Decoding EEG signals of different mental states is a challenging task for brain-computer interfaces (BCIs) due to nonstationarity of perceptual decision processes. This paper presents a novel boosted convolutional neural networks (ConvNets) decoding scheme for motor imagery (MI) EEG signals assisted by the multiwavelet-based time-frequency (TF) causality analysis. Specifically, multiwavelet basis functions are first combined with Geweke spectral measure to obtain high-resolution TF-conditional Granger causality (CGC) representations, where a regularized orthogonal forward regression (ROFR) alg…
3D fluorescence microscopy of living organisms has increasingly become an essential and powerful tool in biomedical research and diagnosis. An exploding amount of imaging data has been collected, whereas efficient and effective computational tools to extract information from them are still lagging behind. This is largely due to the challenges in analyzing biological data. Interesting biological structures are not only small, but are often morphologically irregular and highly dynamic. Although tracking cells in live organisms has been studied for years, existing tracking methods for cells are n…
Understanding Representation Gaps Across Scales in Tropical Tree Species Classification from Drone Imagery
Mila – Quebec AI Institute（魁北克人工智能研究所）；McGill University（麦吉尔大学）；Université de Montréal（蒙特利尔大学）；Cary Institute of Ecosystem Studies（生态系统研究所）；Smithsonian Tropical Research Institute（史密松国家热带研究所）；Department of Plant Sciences, University of Cambridge（剑桥大学植物科学系）；Universidade do Estado do Mato Grosso (UNEMAT)（马托格罗索州立大学（UNEMAT））
Center for Technologies in Robotics and Mechatronics Components（机器人与机电一体化组件技术中心）；Innopolis University（伊诺波利斯大学）；Institute of Robotics（机器人研究所）
Building detection from satellite multispectral imagery data is being a fundamental but a challenging problem mainly because it requires correct recovery of building footprints from high-resolution images. In this work, we propose a deep learning approach for building detection by applying numerous enhancements throughout the process. Initial dataset is preprocessed by 2-sigma percentile normalization. Then data preparation includes ensemble modelling where 3 models were created while incorporating OpenStreetMap data. Binary Distance Transformation (BDT) is used for improving data labeling pro…
Univ. of Almería（阿尔梅里亚大学）；Univ. of Granada（格拉纳达大学）
There is a growing demand for accurate high-resolution land cover maps in many fields, e.g., in land-use planning and biodiversity conservation. Developing such maps has been performed using Object-Based Image Analysis (OBIA) methods, which usually reach good accuracies, but require a high human supervision and the best configuration for one image can hardly be extrapolated to a different image. Recently, the deep learning Convolutional Neural Networks (CNNs) have shown outstanding results in object recognition in the field of computer vision. However, they have not been fully explored yet in …
Automated Linear Disturbance Mapping via Semantic Segmentation of Sentinel-2 Imagery
University of Winnipeg（温尼伯大学）；Hatfield Consultants Group（哈特菲尔德咨询集团）；University of Manitoba（曼尼托巴大学）；Pachama（帕查玛公司）；Environment and Climate Change Canada（加拿大环境与气候变化部）
ReforesTree: A Dataset for Estimating Tropical Forest Carbon Stock with Deep Learning and Aerial Imagery
Technical University of Munich（慕尼黑工业大学）；ETH Zurich（苏黎世联邦理工学院）；Massachusetts Institute of Technology（麻省理工学院）；University of Warwick（华威大学）；New York University（纽约大学）；WWF Switzerland（世界自然基金会瑞士分会）
Time-lapse fluorescence microscopy (TLFM) is an important and powerful tool in synthetic biological research. Modeling TLFM experiments based on real data may enable researchers to repeat certain experiments with minor effort. This thesis is a study towards deep learning-based modeling of TLFM experiments on the image level. The modeling of TLFM experiments, by way of the example of trapped yeast cells, is split into two tasks. The first task is to generate synthetic image data based on real image data. To approach this problem, a novel generative adversarial network, for conditionalized and u…
CIRAD（法国国际发展农业研究中心）；University of Montpellier（蒙彼利埃大学）；AgroParisTech（巴黎高科农业学院）；CNRS（法国国家科学研究中心）；IRSTEA（法国国家环境与农业科学技术研究院）；LIRMM（蒙彼利埃计算机、机器人与微电子实验室）
Nowadays, Earth Observation systems provide a multitude of heterogeneous remote sensing data. How to manage such richness leveraging its complementarity is a crucial chal- lenge in modern remote sensing analysis. Data Fusion techniques deal with this point proposing method to combine and exploit complementarity among the different data sensors. Considering optical Very High Spatial Resolution (VHSR) images, satellites obtain both Multi Spectral (MS) and panchro- matic (PAN) images at different spatial resolution. VHSR images are extensively exploited to produce land cover maps to deal with agr…
