The Algonauts challenge requires to construct a multi-subject encoder of images to brain activity. Deep networks such as ResNet-50 and AlexNet trained for image classification are known to produce feature representations along their intermediate stages which closely mimic the visual hierarchy. However the challenges introduced in the Algonauts project, including combining data from multiple subjects, relying on very few similarity data points, solving for various ROIs, and multi-modality, require devising a flexible framework which can efficiently accommodate them. Here we build upon a recent state-of-the-art classification network (SE-ResNeXt-50) and construct an adaptive combination of its intermediate representations. While the pretrained network serves as a backbone of our model, we learn how to aggregate feature representations along five stages of the network. During learning, our method enables to modulate and screen outputs from each stage along the network as governed by the optimized objective. We applied our method to the Algonauts2019 fMRI and MEG challenges. Using the combined fMRI and MEG data, our approach was rated among the leading five for both challenges. Surprisingly we find that for both the lower and higher order areas (EVC and IT) the adaptive aggregation favors features stemming at later stages of the network.
Aggregate channel features for multi-view face detection
Center for Biometrics and Security Research & National Laboratory of Pattern Recognition(生物识别与安全研究中心暨模式识别国家重点实验室);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Efficient Human Pose Estimation by Learning Deeply Aggregated Representations
Megvii Inc(旷视科技);University of Chinese Academy of Sciences(中国科学院大学);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);Center for Research on Intelligent Perception and Computing(智能感知与计算研究中心);National Laboratory of Pattern Recognition(模式识别国家重点实验室);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
A Linked Aggregate Code for Processing Faces (Revised Version)
Human Information Processing Laboratories(人类信息处理实验室);Advanced Telecommunications Research International(国际先进电信研究所);Laboratory of Vision Research(视觉研究实验室);Rutgers University(罗格斯大学)
3D PersonVLAD: Learning Deep Global Representations for Video-based Person Re-identification
The University of Queensland(昆士兰大学);Hefei University of Technology(合肥工业大学);Dalian University of Technology(大连理工大学);Inception Institute of Artificial Intelligence (IIAI)(人工智能初始研究院)
Neural varifolds: an aggregate representation for quantifying the geometry of point clouds
School of Electronics and Computer Science, University of Southampton(南安普顿大学电子与计算机科学学院);Department of Applied Mathematics and Theoretical Physics, University of Cambridge(剑桥大学应用数学与理论物理系);Institut Camille Jordan, Université Claude Bernard Lyon 1(里昂第一大学卡米尔·约旦研究所)
Learning to Aggregate Multi-Scale Context for Instance Segmentation in Remote Sensing Images
The Hong Kong Polytechnic University(香港理工大学);Wuhan University(武汉大学);Changjiang Spatial Information Technology Engineering Co., Ltd.(长江空间信息技术工程有限公司);Peng Cheng Laboratory(鹏城实验室)
UAGLNet: Uncertainty-Aggregated Global-Local Fusion Network with Cooperative CNN-Transformer for Building Extraction
School of Computer Science (National Pilot Software Engineering School), Beijing University Of Posts and Telecommunications(计算机学院(国家级试点软件工程学院),北京邮电大学);School of Cyber Science and Technology, Shenzhen Campus, Sun Yat-sen University(网络科学与技术学院,中山大学深圳校区)
PAIQ: Patch-Aligned Semantic Injection via Residual Rotation
Tsinghua University(清华大学);Beijing University of Posts and Telecommunications(北京邮电大学);National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学)