Today's computer vision models achieve human or near-human level performance across a wide variety of vision tasks. However, their architectures, data, and learning algorithms differ in numerous ways from those that give rise to human vision. In this paper, we investigate the factors that affect the alignment between the representations learned by neural networks and human mental representations inferred from behavioral responses. We find that model scale and architecture have essentially no effect on the alignment with human behavioral responses, whereas the training dataset and objective function both have a much larger impact. These findings are consistent across three datasets of human similarity judgments collected using two different tasks. Linear transformations of neural network representations learned from behavioral responses from one dataset substantially improve alignment with human similarity judgments on the other two datasets. In addition, we find that some human concepts such as food and animals are well-represented by neural networks whereas others such as royal or sports-related objects are not. Overall, although models trained on larger, more diverse datasets achieve better alignment with humans than models trained on ImageNet alone, our results indicate that scaling alone is unlikely to be sufficient to train neural networks with conceptual representations that match those used by humans.
Evaluating alignment between humans and neural network representations in image-based learning tasks
Institute for Human-Centered AI, Helmholtz Computational Health Center（亥姆霍兹计算健康中心以人为本人工智能研究所）；Max Planck Institute for Biological Cybernetics（马克斯·普朗克生物控制论研究所）；Max Planck School of Cognition（马克斯·普朗克认知科学学院）；Max Planck Institute for Human Cognitive & Brain Sciences（马克斯·普朗克人类认知与脑科学研究所）；University of Tübingen（蒂宾根大学）；Kavli Institute for Systems Neuroscience（卡夫利系统神经科学研究所）；Leipzig University（莱比锡大学）；Technical University Dresden（德累斯顿工业大学）；Julius-Maximilians-Universität Würzburg（维尔茨堡大学）；Max Planck Institute for Human Development（马克斯·普朗克人类发展研究所）
Dimensions underlying the representational alignment of deep neural networks with humans
Max Planck Institute for Human Cognitive and Brain Sciences（马克斯·普朗克人类认知与脑科学研究所）；Donders Institute for Brain, Cognition and Behaviour（唐德斯脑、认知与行为研究所）；Technische Universität Berlin（柏林工业大学）；Berlin Institute for the Foundations of Learning and Data (BIFOLD)（柏林学习与数据基础研究所）；Justus Liebig University（尤斯图斯·李比希大学）；Center for Mind, Brain and Behavior, Universities of Marburg, Giessen, and Darmstadt（马尔堡、吉森和达姆施塔特大学心智、大脑与行为中心）；Google DeepMind（谷歌DeepMind）
CLHA: A Simple yet Effective Contrastive Learning Framework for Human Alignment
University of Science and Technology of China（中国科学技术大学）；Southern University of Science and Technology（南方科技大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen MSU-BIT University（深圳北理莫斯科大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
为解决RLHF训练复杂困难的问题，提出CLHA框架，通过重新评分策略动态调整训练过程，并结合成对对比损失与自适应监督微调损失直接对齐LLMs与人类偏好，在“Helpful and Harmless”数据集上表现卓越。
Achieving More Human Brain-Like Vision via Human EEG Representational Alignment
Department of Psychology, The Ohio State University（心理学系，俄亥俄州立大学）；McGovern Institute for Brain Research, Massachusetts Institute of Technology（脑科学研究所，麻省理工学院）；Department of Neuroscience, The University of Texas at Dallas（神经科学系，德克萨斯大学达拉斯分校）
Do Vision Transformers See Like Humans? Evaluating their Perceptual Alignment
Image Processing Lab, Universidad de Valencia（瓦伦西亚大学图像处理实验室）；Center for Biomaterials and Tissue Engineering Universitat Politecnica de Valencia（多尔多涅理工大学生物材料与组织工程中心）
VisAlign: Dataset for Measuring the Degree of Alignment between AI and Humans in Visual Perception
KAIST（韩国科学技术院）；Seoul National University Bundang Hospital（首尔国立大学盆唐医院）；Seoul National University（首尔国立大学）；MIT（麻省理工学院）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Kangwon National University Hospital（江原国立大学医院）
LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment
Fudan University（复旦大学）；Shanghai Academy of Artificial Intelligence for Science（上海科学人工智能研究院）；Australian Institute for Machine Learning, The University of Adelaide（阿德莱德大学澳大利亚机器学习研究院）
On the dynamic evolution of CLIP texture-shape bias and its relationship to human alignment and model robustness
Image Processing Lab, Universitat de Valencia, Spain（瓦伦西亚大学图像处理实验室）；Centro de Biomateriales e Ingenieria Tisular, Universitat Politecnica de Valencia, Spain（瓦伦西亚理工大学生物材料与组织工程中心）；Computer Vision Center, Spain（西班牙计算机视觉中心）；Universitat Autònoma de Barcelona, Spain（巴塞罗那自治大学）
