The study of the visual system of the brain has attracted the attention and interest of many neuro-scientists, that derived computational models of some types of neuron that compose it. These findings inspired researchers in image processing and computer vision to deploy such models to solve problems of visual data processing. In this paper, we review approaches for image processing and computer vision, the design of which is based on neuro-scientific findings about the functions of some neurons in the visual cortex. Furthermore, we analyze the connection between the hierarchical organization of the visual system of the brain and the structure of Convolutional Networks (ConvNets). We pay particular attention to the mechanisms of inhibition of the responses of some neurons, which provide the visual system with improved stability to changing input stimuli, and discuss their implementation in image processing operators and in ConvNets.
Department of Electrical Engineering and Computer Science, University of Arkansas（电气工程与计算机科学系，阿肯色大学）；Department of Physics, University of Arkansas（物理系，阿肯色大学）；Department of Radiology, SUNY Upstate Medical University（放射学系，SUNY Upstate医学大学）；Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology（脑科学与认知科学系，麻省理工学院）；Department of Electrical and Computer Engineering, Kansas State University（电气与计算机工程系，堪萨斯州立大学）
Vision-brain understanding aims to extract semantic information about brain signals from human perceptions. Existing deep learning methods for vision-brain understanding are usually introduced in a traditional learning paradigm missing the ability to learn the connectivities between brain regions. Meanwhile, the quantum computing theory offers a new paradigm for designing deep learning models. Motivated by the connectivities in the brain signals and the entanglement properties in quantum computing, we propose a novel Quantum-Brain approach, a quantum-inspired neural network, to tackle the visi…
Brain-inspired analogical mixture prototypes for few-shot class-incremental learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院）
Northwestern Polytechnical University（西北工业大学）；University of Georgia（佐治亚大学）；The University of Texas at Arlington（德克萨斯大学阿灵顿分校）
Visual attention is a fundamental mechanism in the human brain, and it inspires the design of attention mechanisms in deep neural networks. However, most of the visual attention studies adopted eye-tracking data rather than the direct measurement of brain activity to characterize human visual attention. In addition, the adversarial relationship between the attention-related objects and attention-neglected background in the human visual system was not fully exploited. To bridge these gaps, we propose a novel brain-inspired adversarial visual attention network (BI-AVAN) to characterize human vis…
Key Laboratory for Radio Astronomy, Chinese Academy of Sciences（中国科学院射电天文重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）
Evolution of deep learning shows that some algorithmic tricks are more durable , while others are not. To the best of our knowledge, we firstly summarize 5 more durable and complete deep learning components for vision, that is, WARSHIP. Moreover, we give a biological overview of WARSHIP, emphasizing brain-inspired computing of WARSHIP. As a step towards WARSHIP, our case study of image super resolution combines 3 components of RSH to deploy a CNN model of WARSHIP-XZNet, which performs a happy medium between speed and performance.
ShanghaiTech University（上海科技大学）；Shanghai Center for Brain Science and Brain-Inspired Technology（上海脑科学与类脑研究中心）；Tsinghua University（清华大学）；University of Science and Technology of China（中国科学技术大学）；University of Shanghai for Science and Technology（上海理工大学）
Despite its great success, deep learning severely suffers from robustness; that is, deep neural networks are very vulnerable to adversarial attacks, even the simplest ones. Inspired by recent advances in brain science, we propose the Denoised Internal Models (DIM), a novel generative autoencoder-based model to tackle this challenge. Simulating the pipeline in the human brain for visual signal processing, DIM adopts a two-stage approach. In the first stage, DIM uses a denoiser to reduce the noise and the dimensions of inputs, reflecting the information pre-processing in the thalamus. Inspired f…
Recently, Multilayer Perceptron (MLP) becomes the hotspot in the field of computer vision tasks. Without inductive bias, MLPs perform well on feature extraction and achieve amazing results. However, due to the simplicity of their structures, the performance highly depends on the local features communication machenism. To further improve the performance of MLP, we introduce information communication mechanisms from brain-inspired neural networks. Spiking Neural Network (SNN) is the most famous brain-inspired neural network, and achieve great success on dealing with sparse data. Leaky Integrate …
Quantitative modeling of human brain activity based on language representations has been actively studied in systems neuroscience. However, previous studies examined word-level representation, and little is known about whether we could recover structured sentences from brain activity. This study attempts to generate natural language descriptions of semantic contents from human brain activity evoked by visual stimuli. To effectively use a small amount of available brain activity data, our proposed method employs a pre-trained image-captioning network model using a deep learning framework. To ap…
Rochester Institute of Technology（罗切斯特理工学院）；Carlson Center for Imaging Science（卡尔森成像科学中心）
Incremental class learning involves sequentially learning classes in bursts of examples from the same class. This violates the assumptions that underlie methods for training standard deep neural networks, and will cause them to suffer from catastrophic forgetting. Arguably, the best method for incremental class learning is iCaRL, but it requires storing training examples for each class, making it challenging to scale. Here, we propose FearNet for incremental class learning. FearNet is a generative model that does not store previous examples, making it memory efficient. FearNet uses a brain-ins…
Beckman Institute（贝克曼研究所）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Image aesthetics assessment has been challenging due to its subjective nature. Inspired by the scientific advances in the human visual perception and neuroaesthetics, we design Brain-Inspired Deep Networks (BDN) for this task. BDN first learns attributes through the parallel supervised pathways, on a variety of selected feature dimensions. A high-level synthesis network is trained to associate and transform those attributes into the overall aesthetics rating. We then extend BDN to predicting the distribution of human ratings, since aesthetics ratings are often subjective. Another highlight is …
Although deep learning has solved difficult problems in visual pattern recognition, it is mostly successful in tasks where there are lots of labeled training data available. Furthermore, the global back-propagation based training rule and the amount of employed layers represents a departure from biological inspiration. The brain is able to perform most of these tasks in a very general way from limited to no labeled data. For these reasons it is still a key research question to look into computational principles in the brain that can help guide models to unsupervisedly learn good representation…
State Key Laboratory of Media Convergence and Communication, Communication University of China（媒体融合与传播国家重点实验室，中国传媒大学）；Neuroscience and Intelligent Media Institute, Communication University of China（神经科学与智能媒体研究院，中国传媒大学）
The partially occluded image recognition (POIR) problem has been a challenge for artificial intelligence for a long time. A common strategy to handle the POIR problem is using the non-occluded features for classification. Unfortunately, this strategy will lose effectiveness when the image is severely occluded, since the visible parts can only provide limited information. Several studies in neuroscience reveal that feature restoration which fills in the occluded information and is called amodal completion is essential for human brains to recognize partially occluded images. However, feature res…
Isfahan University of Technology（伊斯法罕理工大学）；McMaster University（麦克马斯特大学）；University of Michigan（密歇根大学）
The use of 3D and stereo imaging is rapidly increasing. Compression, transmission, and processing could degrade the quality of stereo images. Quality assessment of such images is different than their 2D counterparts. Metrics that represent 3D perception by human visual system (HVS) are expected to assess stereoscopic quality more accurately. In this paper, inspired by brain sensory/motor fusion process, two stereo images are fused together. Then from every fused image two synthesized images are extracted. Effects of different distortions on statistical distributions of the synthesized images a…
This paper presents a new proposal of an efficient computational model of face recognition which uses cues from the distributed face recognition mechanism of the brain, and by gathering engineering equivalent of these cues from existing literature. Three distinct and widely used features: Histogram of Oriented Gradients (HOG), Local Binary Patterns (LBP), and Principal components (PCs) extracted from target images are used in a manner which is simple, and yet effective. The HOG and LBP features further undergo principal component analysis for dimensionality reduction. Our model uses multi-laye…
Recent machine learning models have shown that including attention as a component results in improved model accuracy and interpretability, despite the concept of attention in these approaches only loosely approximating the brain's attention mechanism. Here we extend this work by building a more brain-inspired deep network model of the primate ATTention Network (ATTNet) that learns to shift its attention so as to maximize the reward. Using deep reinforcement learning, ATTNet learned to shift its attention to the visual features of a target category in the context of a search task. ATTNet's dors…
Bar Ilan University（巴伊兰大学）；Tel Aviv University（特拉维夫大学）
Sparse-to-dense interpolation for optical flow is a fundamental phase in the pipeline of most of the leading optical flow estimation algorithms. The current state-of-the-art method for interpolation, EpicFlow, is a local average method based on an edge aware geodesic distance. We propose a new data-driven sparse-to-dense interpolation algorithm based on a fully convolutional network. We draw inspiration from the filling-in process in the visual cortex and introduce lateral dependencies between neurons and multi-layer supervision into our learning process. We also show the importance of the ima…
BRACTIVE: A Brain Activation Approach to Human Visual Brain Learning
Electrical Engineering & Computer Science Department, University of Arkansas（亚拉巴马大学电气工程与计算机科学系）；Brain and Cognitive Engineering, Korea University（韩国大学脑科学与认知工程系）；Department of Computer Science, University at Albany（阿尔巴尼大学计算机科学系）；Mike Wiegers Department of Electrical & Computer Engineering, Kansas State University（堪萨斯州立大学迈克·魏格斯电气与计算机工程系）；Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology（麻省理工学院脑科学与认知科学系）
Brain-Inspired Multimodal Spiking Neural Network for Image-Text Retrieval
Hubei Key Laboratory of Transportation Internet of Things, Wuhan University of Technology（武汉理工大学交通物联网湖北省重点实验室）；State Key Laboratory for Multimedia Information Processing, Peking University（北京大学多媒体信息处理全国重点实验室）
MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Nanjing University of Science and Technology（南京理工大学）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）
School of Electronics and Computer Engineering, Peking University（北京大学电子与计算机工程学院）
In this paper, we propose a novel end-to-end architecture that could generate a variety of plausible video sequences correlating two given discontinuous frames. Our work is inspired by the human ability of inference. Specifically, given two static images, human are capable of inferring what might happen in between as well as present diverse versions of their inference. We firstly train our model to learn the transformation to understand the movement trends within given frames. For the sake of imitating the inference of human, we introduce a latent variable sampled from Gaussian distribution. B…
The new perspective in visual classification aims to decode the feature representation of visual objects from human brain activities. Recording electroencephalogram (EEG) from the brain cortex has been seen as a prevalent approach to understand the cognition process of an image classification task. In this study, we proposed a deep learning framework guided by the visual evoked potentials, called the Event-Related Potential (ERP)-Long short-term memory (LSTM) framework, extracted by EEG signals for visual classification. In specific, we first extracted the ERP sequences from multiple EEG chann…
Bernoulli Institute, University of Groningen（格罗宁根大学伯努利研究所）
In this paper we present a novel filter, based on the existing COSFIRE filter, for the delineation of patterns of interest. It includes a mechanism of push-pull inhibition that improves robustness to noise in terms of spurious texture. Push-pull inhibition is a phenomenon that is observed in neurons in area V1 of the visual cortex, which suppresses the response of certain simple cells for stimuli of preferred orientation but of non-preferred contrast. This type of inhibition allows for sharper detection of the patterns of interest and improves the quality of delineation especially in images wi…
CerCo CNRS（法国国家科学研究中心大脑与认知研究中心）；Université de Toulouse（图卢兹大学）；IRIT CNRS（法国国家科学研究中心图卢兹计算机科学研究所）；ANITI（图卢兹人工智能跨学科研究所）
Deep neural networks excel at image classification, but their performance is far less robust to input perturbations than human perception. In this work we explore whether this shortcoming may be partly addressed by incorporating brain-inspired recurrent dynamics in deep convolutional networks. We take inspiration from a popular framework in neuroscience: 'predictive coding'. At each layer of the hierarchical model, generative feedback 'predicts' (i.e., reconstructs) the pattern of activity in the previous layer. The reconstruction errors are used to iteratively update the network's representat…
University of Catania（卡塔尼亚大学）；University of Central Florida（中佛罗里达大学）
This work presents a novel method of exploring human brain-visual representations, with a view towards replicating these processes in machines. The core idea is to learn plausible computational and biological representations by correlating human neural activity and natural images. Thus, we first propose a model, EEG-ChannelNet, to learn a brain manifold for EEG classification. After verifying that visual information can be extracted from EEG data, we introduce a multimodal approach that uses deep image and EEG encoders, trained in a siamese configuration, for learning a joint manifold that max…
Brain-Inspired Capture: Evidence-Driven Neuromimetic Perceptual Simulation for Visual Decoding
School of Mathematics, Taiyuan University of Technology（太原科技大学数学学院）；College of Artificial Intelligence, Taiyuan University of Technology（太原科技大学人工智能学院）；National Human Genetics Resource Center, National Research Institute for Family Planning（国家人类遗传资源中心、国家计划生育研究院）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）
Tongji University（同济大学）；University of Oxford（牛津大学）；North China Institute of Computing Technology, State Information Center of China（国家信息中心华北计算技术研究所）；Macquarie University（麦考瑞大学）
Brain-SAD: A Brain-Inspired Safe Autonomous Driving Control Framework with Dynamic Fear-Oriented Constraint on Dual-Policy
Nanjing University of Information Science & Technology（南京信息工程大学）；University of Electronic Science and Technology of China（电子科技大学）；Jiangsu Ocean University（江苏海洋大学）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Ningbo HwaMei Hospital, UCAS（中国科学院大学宁波华美医院）
Decoding human visual neural representations is a challenging task with great scientific significance in revealing vision-processing mechanisms and developing brain-like intelligent machines. Most existing methods are difficult to generalize to novel categories that have no corresponding neural data for training. The two main reasons are 1) the under-exploitation of the multimodal semantic knowledge underlying the neural data and 2) the small number of paired (stimuli-responses) training data. To overcome these limitations, this paper presents a generic neural decoding method called BraVL that…
University of California Berkeley（加州大学伯克利分校）；Helen Wills Neuroscience Institute（海伦·威尔斯神经科学研究所）
The human brain is adept at solving difficult high-level visual processing problems such as image interpretation and object recognition in natural scenes. Over the past few years neuroscientists have made remarkable progress in understanding how the human brain represents categories of objects and actions in natural scenes. However, all current models of high-level human vision operate on hand annotated images in which the objects and actions have been assigned semantic tags by a human operator. No current models can account for high-level visual function directly in terms of low-level visual …
Enabling effective brain-computer interfaces requires understanding how the human brain encodes stimuli across modalities such as visual, language (or text), etc. Brain encoding aims at constructing fMRI brain activity given a stimulus. There exists a plethora of neural encoding models which study brain encoding for single mode stimuli: visual (pretrained CNNs) or text (pretrained language models). Few recent papers have also obtained separate visual and text representation models and performed late-fusion using simple heuristics. However, previous work has failed to explore: (a) the effective…
