Humans effortlessly navigate the dynamic visual world, yet deep neural networks (DNNs), despite excelling at many visual tasks, are surprisingly vulnerable to minor image perturbations. Past theories suggest that human visual robustness arises from a representational space that evolves along the ventral visual stream (VVS) of the brain to increasingly tolerate object transformations. To test whether robustness is supported by such progression as opposed to being confined exclusively to specialized higher-order regions, we trained DNNs to align their representations with human neural responses from consecutive VVS regions while performing visual tasks. We demonstrate a hierarchical improvement in DNN robustness: alignment to higher-order VVS regions leads to greater improvement. To investigate the mechanism behind such robustness gains, we test a prominent hypothesis that attributes human robustness to the unique geometry of neural category manifolds in the VVS. We first reveal that more desirable manifold properties, specifically, smaller extent and better linear separability, indeed emerge across the human VVS. These properties can be inherited by neurally aligned DNNs and predict their subsequent robustness gains. Furthermore, we show that supervision from neural manifolds alone, via manifold guidance, is sufficient to qualitatively reproduce the hierarchical robustness improvements. Together, these results highlight the critical role of the evolving representational space across VVS in achieving robust visual inference, in part through the formation of more linearly separable category manifolds, which may in turn be leveraged to develop more robust AI systems.
FakeRadar: Probing Forgery Outliers to Detect Unknown Deepfake Videos
Guilin University of Electronic Technology(桂林电子科技大学);Pengcheng Laboratory(鹏城实验室);Sun Yat-sen University(中山大学);Guangxi Key Laboratory of Image and Graphic Intelligent Processing(广西图像与图形智能处理重点实验室);Guangdong Key Laboratory of Big Data Analysis and Processing(广东省大数据分析与处理重点实验室)
Audio-Guided Fusion Techniques for Multimodal Emotion Analysis
Beijing University of Posts and Telecommunications(北京邮电大学);State Key Laboratory of Networking and Switching Technology(网络与交换技术国家重点实验室);School of Cyberspace Security(网络空间安全学院)
Beyond Image-Text Matching: Verb Understanding in Multimodal Transformers Using Guided Masking
Faculty of Information Technology, Brno University of Technology(布尔诺理工大学信息技术学院);Kempelen Institute of Intelligent Technologies(肯佩伦智能技术研究所);George Mason University(乔治梅森大学)
ReMAP-PET: Beyond Visual Understanding -- Learning Region-Guided Metabolic Alignment Semantics from Brain PET
The Chinese University of Hong Kong, HKSAR(香港中文大学);TReNDS Center (Georgia State, Georgia Tech, Emory)(TReNDS中心(佐治亚州立大学、佐治亚理工学院、埃默里大学));ShanghaiTech University, Shanghai, P.R.China(上海科技大学);University of California, Berkeley, USA(加州大学伯克利分校);University of Texas Health Science Center at Houston, USA(德克萨斯大学健康科学中心(休斯顿));Nanyang Technological University, Singapore(南洋理工大学)
HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes
Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, China(中国科学院智能信息处理重点实验室(中国科学院)、计算技术研究所、中国)
Learnable Shape Prototypes with Occlusion-Geometry-Guided Injection for Amodal Instance Segmentation
School of Mechanical Engineering and Automation, Northeastern University(机械工程与自动化学院,东北大学);School of Information Science and Engineering, Northeastern University(信息科学与工程学院,东北大学)