In neuroscience, understanding inter-individual differences has recently emerged as a major challenge, for which functional magnetic resonance imaging (fMRI) has proven invaluable. For this, neuroscientists rely on basic methods such as univariate linear correlations between single brain features and a score that quantifies either the severity of a disease or the subject's performance in a cognitive task. However, to this date, task-fMRI and resting-state fMRI have been exploited separately for this question, because of the lack of methods to effectively combine them. In this paper, we introduce a novel machine learning method which allows combining the activation-and connectivity-based information respectively measured through these two fMRI protocols to identify markers of individual differences in the functional organization of the brain. It combines a multi-view deep autoencoder which is designed to fuse the two fMRI modalities into a joint representation space within which a predictive model is trained to guess a scalar score that characterizes the patient. Our experimental results demonstrate the ability of the proposed method to outperform competitive approaches and to produce interpretable and biologically plausible results.
Identification of morphological fingerprint in perinatal brains using quasi-conformal mapping and contrastive learning
Lanzhou University（兰州大学）；Zhejiang University（浙江大学）；Hangzhou Normal University（杭州师范大学）；Lanzhou Jiaotong University（兰州交通大学）；Binjiang Institute of Zhejiang University（浙江大学滨江研究院）
CSIRO（联邦科学与工业研究组织）；Queensland University of Technology（昆士兰科技大学）；University of Queensland（昆士兰大学）
The problem of Cortical Surface Reconstruction from magnetic resonance imaging has been traditionally addressed using lengthy pipelines of image processing techniques like FreeSurfer, CAT, or CIVET. These frameworks require very long runtimes deemed unfeasible for real-time applications and unpractical for large-scale studies. Recently, supervised deep learning approaches have been introduced to speed up this task cutting down the reconstruction time from hours to seconds. Using the state-of-the-art CorticalFlow model as a blueprint, this paper proposes three modifications to improve its accur…
Individualized Mapping of Aberrant Cortical Thickness via Stochastic Cortical Self-Reconstruction
Lab for Artificial Intelligence in Medical Imaging, Institute for Diagnostic and Interventional Radiology, School of Medicine and Health, TUM Klinikum, Technical University of Munich (TUM)（人工智能在医学影像中的实验室、诊断与介入放射学研究所、医学与健康学院、TUM医院、慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Department of Neuroradiology, School of Medicine and Health, TUM Klinikum, Technical University of Munich (TUM)（神经放射科、医学与健康学院、TUM医院、慕尼黑技术大学）
Learning Cortical Anomaly through Masked Encoding for Unsupervised Heterogeneity Mapping
Department of Psychiatry and Psychotherapy, Tübingen Center for Mental Health (TÜCMH)（图宾根心理健康中心精神病学与心理治疗系）；Norwegian Centre for Mental Disorders Research（挪威精神障碍研究中心）；Donders Institute for Brain, Cognition, and Behaviour, Radboud University（拉德堡德大学唐德斯脑、认知与行为研究所）
Why all roads don't lead to Rome: Representation geometry varies across the human visual cortical hierarchy
Mila - Quebec AI Institute（魁北克人工智能研究所）；Computer Science, McGill University（麦吉尔大学计算机科学系）；Department of Psychology, Université de Montréal（蒙特利尔大学心理学系）；Department of Neurology & Neurosurgery, McGill University（麦吉尔大学神经学与神经外科系）；Montreal Neurological Institute（蒙特利尔神经科学研究所）；CIFAR Learning in Machines & Brains Program（CIFAR机器与大脑学习计划）
The ability to efficiently utilize crowdsourced visual data carries immense potential for the domains of large scale dynamic mapping and autonomous driving. However, state-of-the-art methods for crowdsourced 3D mapping assume prior knowledge of camera intrinsics. In this work, we propose a framework that estimates the 3D positions of semantically meaningful landmarks such as traffic signs without assuming known camera intrinsics, using only monocular color camera and GPS. We utilize multi-view geometry as well as deep learning based self-calibration, depth, and ego-motion estimation for traffi…
Institute of Neuroscience and Medicine (INM-1), Research Centre Jülich（于利希研究中心神经科学与医学研究所（INM-1））；Helmholtz AI, Research Centre Jülich（于利希研究中心亥姆霍兹人工智能）；Heinrich Heine University, Düsseldorf（杜塞尔多夫海因里希·海涅大学）；Cécile & Oscar Vogt Institute for Brain Research, University Hospital Düsseldorf（杜塞尔多夫大学医院塞西尔和奥斯卡·沃格特脑研究所）
Cytoarchitecture describes the spatial organization of neuronal cells in the brain, including their arrangement into layers and columns with respect to cell density, orientation, or presence of certain cell types. It allows to segregate the brain into cortical areas and subcortical nuclei, links structure with connectivity and function, and provides a microstructural reference for human brain atlases. Mapping boundaries between areas requires to scan histological sections at microscopic resolution. While recent high-throughput scanners allow to scan a complete human brain in the order of a yea…
Northwestern Polytechnical University（西北工业大学）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）；University of Georgia（佐治亚大学）
Since gyri and sulci, two basic anatomical building blocks of cortical folding patterns, were suggested to bear different functional roles, a precise mapping from brain function to gyro-sulcal patterns can provide profound insights into both biological and artificial neural networks. However, there lacks a generic theory and effective computational model so far, due to the highly nonlinear relation between them, huge inter-individual variabilities and a sophisticated description of brain function regions/networks distribution as mosaics, such that spatial patterning of them has not been consid…
From Low Field to High Value: Robust Cortical Mapping from Low-Field MRI
Massachusetts General Hospital, Harvard Medical School, United States（哈佛医学院麻省总医院，美国）；Massachusetts Institute of Technology, United States（麻省理工学院，美国）；University of Washington, United States（华盛顿大学，美国）；Danish Research Centre for Magnetic Resonance, Copenhagen University Hospital, Denmark（丹麦磁共振研究中心，哥本哈根大学医院，丹麦）；Washington University, United States（华盛顿大学，美国）；University College London, United Kingdom（伦敦大学学院，英国）
The Chinese University of Hong Kong（香港中文大学）；Massachusetts Institute of Technology（麻省理工学院）
In medical imaging, surface registration is extensively used for performing systematic comparisons between anatomical structures, with a prime example being the highly convoluted brain cortical surfaces. To obtain a meaningful registration, a common approach is to identify prominent features on the surfaces and establish a low-distortion mapping between them with the feature correspondence encoded as landmark constraints. Prior registration works have primarily focused on using manually labeled landmarks and solving highly nonlinear optimization problems, which are time-consuming and hence hin…
School of Computing, Informatics and Decision Systems Engineering, Arizona State University（亚利桑那州立大学计算、信息与决策系统工程学院）
A large number of surface-based analyses on brain imaging data adopt some specific brain atlases to better assess structural and functional changes in one or more brain regions. In these analyses, it is necessary to obtain an anatomically correct surface parcellation scheme in an individual brain by referring to the given atlas. Traditional ways to accomplish this goal are through a designed surface-based registration or hand-crafted surface features, although both of them are time-consuming. A recent deep learning approach depends on a regular spherical parameterization of the mesh, which is …
Shared representations in brains and models reveal a two-route cortical organization during scene perception
Department of Cognition, Development and Education Psychology, Faculty of Psychology, University of Barcelona（认知、发展与教育心理学系，心理学学院，巴塞罗那大学）；Institute of Neurosciences, University of Barcelona（神经科学研究所，巴塞罗那大学）；Bellvitge Institute for Biomedical Research（Bellvitge生物医学研究 institute）
Massachusetts Institute of Technology（麻省理工学院）；Free University Berlin（柏林自由大学）
The complex multi-stage architecture of cortical visual pathways provides the neural basis for efficient visual object recognition in humans. However, the stage-wise computations therein remain poorly understood. Here, we compared temporal (magnetoencephalography) and spatial (functional MRI) visual brain representations with representations in an artificial deep neural network (DNN) tuned to the statistics of real-world visual recognition. We showed that the DNN captured the stages of human visual processing in both time and space from early visual areas towards the dorsal and ventral streams…
Improved Multiscale Structural Mapping with Supervertex Vision Transformer for the Detection of Alzheimer's Disease Neurodegeneration
Department of Computer Science \& Engineering, Hankuk University of Foreign Studies, Seoul, Republic of Korea；Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Charlestown, MA, USA；Department of Radiology, Harvard Medical School, Boston, MA, USA；Neuroimaging Research for Veterans (NeRVe) Center, VA Boston Healthcare System, Boston, MA, USA
Community-Level Modeling of Gyral Folding Patterns for Robust and Anatomically Informed Individualized Brain Mapping
Department of Computer Science and Engineering, University of Texas at Arlington（计算机科学与工程系，德克萨斯大学阿灵顿分校）；School of Computing, University of Georgia（计算学院，佐治亚大学）；Department of Computer Science, Indiana University Indianapolis（计算机科学系，印第安纳大学印第安纳波利斯分校）
Google（谷歌）；National University of Singapore（新加坡国立大学）
Mapping and localization, preferably from a small number of observations, are fundamental tasks in robotics. We address these tasks by combining spatial structure (differentiable mapping) and end-to-end learning in a novel neural network architecture: the Differentiable Mapping Network (DMN). The DMN constructs a spatially structured view-embedding map and uses it for subsequent visual localization with a particle filter. Since the DMN architecture is end-to-end differentiable, we can jointly learn the map representation and localization using gradient descent. We apply the DMN to sparse visua…
The Geometry of Cortical Computation: Manifold Disentanglement and Predictive Dynamics in VCNet
Department of Computer Science University of Wisconsin-Madison（计算机科学系 威斯康星大学麦迪逊分校）；Department of Computer Science National University of Singapore（计算机科学系 新加坡国立大学）
University of California, San Diego（加州大学圣地亚哥分校）；Google Research（谷歌研究院）
Recent work in Neural Fields (NFs) learn 3D representations from class-specific single view image collections. However, they are unable to reconstruct the input data preserving high-frequency details. Further, these methods do not disentangle appearance from geometry and hence are not suitable for tasks such as texture transfer and editing. In this work, we propose TEGLO (Textured EG3D-GLO) for learning 3D representations from single view in-the-wild image collections for a given class of objects. We accomplish this by training a conditional Neural Radiance Field (NeRF) without any explicit 3D…
We propose a weakly-supervised multi-view learning approach to learn category-specific surface mapping without dense annotations. We learn the underlying surface geometry of common categories, such as human faces, cars, and airplanes, given instances from those categories. While traditional approaches solve this problem using extensive supervision in the form of pixel-level annotations, we take advantage of the fact that pixel-level UV and mesh predictions can be combined with 3D reprojections to form consistency cycles. As a result of exploiting these cycles, we can establish a dense correspo…
Tsinghua University（清华大学）；University of Maryland（马里兰大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）
3D shape captioning is a challenging application in 3D shape understanding. Captions from recent multi-view based methods reveal that they cannot capture part-level characteristics of 3D shapes. This leads to a lack of detailed part-level description in captions, which human tend to focus on. To resolve this issue, we propose ShapeCaptioner, a generative caption network, to perform 3D shape captioning from semantic parts detected in multiple views. Our novelty lies in learning the knowledge of part detection in multiple views from 3D shape segmentations and transferring this knowledge to facil…
University of Central Florida（中佛罗里达大学）；Florida Hospital for Children（佛罗里达儿童医院）
Accurate localization of brain regions responsible for language and cognitive functions in Epilepsy patients should be carefully determined prior to surgery. Electrocorticography (ECoG)-based Real Time Functional Mapping (RTFM) has been shown to be a safer alternative to the electrical cortical stimulation mapping (ESM), which is currently the clinical/gold standard. Conventional methods for analyzing RTFM signals are based on statistical comparison of signal power at certain frequency bands. Compared to gold standard (ESM), they have limited accuracies when assessing channel responses. In thi…
Multi-view Feature Augmentation with Adaptive Class Activation Mapping
School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；Research Center on Fictitious Economy and Data Science, Chinese Academy of Sciences（中国科学院虚拟经济与数据科学研究中心）；Key Laboratory of Big Data Mining and Knowledge Management, Chinese Academy of Sciences（中国科学院大数据挖掘与知识管理重点实验室）
Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）；FZI Research Center for Information Technology（FZI信息技术研究中心）
We present a generic evidential grid mapping pipeline designed for imaging sensors such as LiDARs and cameras. Our grid-based evidential model contains semantic estimates for cell occupancy and ground separately. We specify the estimation steps for input data represented by point sets, but mainly focus on input data represented by images such as disparity maps or LiDAR range images. Instead of relying on an external ground segmentation only, we deduce occupancy evidence by analyzing the surface orientation around measurements. We conduct experiments and evaluate the presented method using LiDA…
Ground referencing is essential for supervised crop mapping. However, conventional ground truthing involves extensive field surveys and post processing, which is costly in terms of time and labor. In this study, we applied a convolutional neural network (CNN) model to explore the efficacy of automatic ground truthing via Google street view (GSV) images in two distinct farming regions: central Illinois and southern California. We demonstrated the feasibility and reliability of the new ground referencing technique further by performing pixel-based crop mapping with vegetation indices as the mode…
Visual scene understanding is an important capability that enables robots to purposefully act in their environment. In this paper, we propose a novel approach to object-class segmentation from multiple RGB-D views using deep learning. We train a deep neural network to predict object-class semantics that is consistent from several view points in a semi-supervised way. At test time, the semantics predictions of our network can be fused more consistently in semantic keyframe maps than predictions of a network trained on individual views. We base our network architecture on a recent single-view de…
Technical University of Munich（慕尼黑工业大学）；Artisense；Karlsruhe University of Applied Sciences（卡尔斯鲁厄应用科学大学）
In this paper, we present TANDEM a real-time monocular tracking and dense mapping framework. For pose estimation, TANDEM performs photometric bundle adjustment based on a sliding window of keyframes. To increase the robustness, we propose a novel tracking front-end that performs dense direct image alignment using depth maps rendered from a global model that is built incrementally from dense depth predictions. To predict the dense depth maps, we propose Cascade View-Aggregation MVSNet (CVA-MVSNet) that utilizes the entire active keyframe window by hierarchically constructing 3D cost volumes wit…
Neural field-based 3D representations have recently been adopted in many areas including SLAM systems. Current neural SLAM or online mapping systems lead to impressive results in the presence of simple captures, but they rely on a world-centric map representation as only a single neural field model is used. To define such a world-centric representation, accurate and static prior information about the scene, such as its boundaries and initial camera poses, are required. However, in real-time and on-the-fly scene capture applications, this prior knowledge cannot be assumed as fixed or static, si…
Due to the growing concern of chronic diseases and other health problems related to diet, there is a need to develop accurate methods to estimate an individual's food and energy intake. Measuring accurate dietary intake is an open research problem. In particular, accurate food portion estimation is challenging since the process of food preparation and consumption impose large variations on food shapes and appearances. In this paper, we present a food portion estimation method to estimate food energy (kilocalories) from food images using Generative Adversarial Networks (GAN). We introduce the c…
National Digital Switching System Engineering and Technological Research Center（国家数字交换系统工程技术研究中心）；University of Minnesota at Twin Cities（明尼苏达大学双城分校）
Background: Building visual encoding models to accurately predict visual responses is a central challenge for current vision-based brain-machine interface techniques. To achieve high prediction accuracy on neural signals, visual encoding models should include precise visual features and appropriate prediction algorithms. Most existing visual encoding models employ hand-craft visual features (e.g., Gabor wavelets or semantic labels) or data-driven features (e.g., features extracted from deep neural networks (DNN)). They also assume a linear mapping between feature representation to brain activi…
Humans have an innate ability to sense their surroundings, as they can extract the spatial representation from the egocentric perception and form an allocentric semantic map via spatial transformation and memory updating. However, endowing mobile agents with such a spatial sensing ability is still a challenge, due to two difficulties: (1) the previous convolutional models are limited by the local receptive field, thus, struggling to capture holistic long-range dependencies during observation; (2) the excessive computational budgets required for success, often lead to a separation of the mappin…
O2V-Mapping: Online Open-Vocabulary Mapping with Neural Implicit Representation
Academy for Engineering & Technology（工程与技术学院）；Fudan University（复旦大学）；State Key Laboratory of Intelligent Vehicle Safety Technology（智能车辆安全技术国家重点实验室）；School of Future Technology（未来技术学院）；Harbin Institute of Technology（哈尔滨工业大学）；Department of Computer Science and Engineering（计算机科学与工程系）；Shanghai Jiao Tong University（上海交通大学）
Cross-View Geolocalization and Disaster Mapping with Street-View and VHR Satellite Imagery: A Case Study of Hurricane IAN
Technical University of Munich（慕尼黑工业大学）；University of the Bundeswehr Munich（慕尼黑联邦国防军大学）；China University of Mining and Technology（中国矿业大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Tongji University（同济大学）
In this work, we research and evaluate end-to-end learning of monocular semantic-metric occupancy grid mapping from weak binocular ground truth. The network learns to predict four classes, as well as a camera to bird's eye view mapping. At the core, it utilizes a variational encoder-decoder network that encodes the front-view visual information of the driving scene and subsequently decodes it into a 2-D top-view Cartesian coordinate system. The evaluations on Cityscapes show that the end-to-end learning of semantic-metric occupancy grids outperforms the deterministic mapping approach with flat…
Hi-DREAM: Brain-Inspired Hierarchical Diffusion for fMRI-to-Image Reconstruction via ROI Encoder and VisuAl Mapping
Dept of Data Science and AI, Monash University, Melbourne, Australia（数据科学与人工智能系，墨尔本大学，澳大利亚）；School of Psychological Sciences, Monash University, Melbourne, Australia（心理学科学学院，墨尔本大学，澳大利亚）；Brain and Mind Centre, University of Sydney, Sydney, Australia（脑与心智中心，悉尼大学，澳大利亚）
We present cross-view transformers, an efficient attention-based model for map-view semantic segmentation from multiple cameras. Our architecture implicitly learns a mapping from individual camera views into a canonical map-view representation using a camera-aware cross-view attention mechanism. Each camera uses positional embeddings that depend on its intrinsic and extrinsic calibration. These embeddings allow a transformer to learn the mapping across different views without ever explicitly modeling it geometrically. The architecture consists of a convolutional image encoder for each view and…
University of Freiburg（弗莱堡大学）；Federal University of Rio Grande（里奥格兰德联邦大学）；University of Technology Nuremberg（纽伦堡工业大学）
Bird's-Eye-View (BEV) semantic maps have become an essential component of automated driving pipelines due to the rich representation they provide for decision-making tasks. However, existing approaches for generating these maps still follow a fully supervised training paradigm and hence rely on large amounts of annotated BEV data. In this work, we address this limitation by proposing the first self-supervised approach for generating a BEV semantic map using a single monocular image from the frontal view (FV). During training, we overcome the need for BEV ground truth annotations by leveraging …
