As we consider entrusting Large Language Models (LLMs) with key societal and decision-making roles, measuring their alignment with human cognition becomes critical. This requires methods that can assess how these systems represent information and facilitate comparisons with human understanding across diverse tasks. To meet this need, we adapted Representational Similarity Analysis (RSA), a method that uses pairwise similarity ratings to quantify alignment between AIs and humans. We tested this approach on semantic alignment across text and image modalities, measuring how different Large Language and Vision Language Model (LLM and VLM) similarity judgments aligned with human responses at both group and individual levels. GPT-4o showed the strongest alignment with human performance among the models we tested, particularly when leveraging its text processing capabilities rather than image processing, regardless of the input modality. However, no model we studied adequately captured the inter-individual variability observed among human participants, and only moderately aligned with any individual human's responses. This method helped uncover certain hyperparameters and prompts that could steer model behavior to have more or less human-like qualities at an inter-individual or group level. Pairwise ratings and RSA enable the efficient and flexible quantification of human-AI alignment, which complements existing accuracy-based benchmark tasks. We demonstrate the utility of this approach across multiple modalities (words, sentences, images) for understanding how LLMs encode knowledge and for examining representational alignment with human cognition.
Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It
Boston University（波士顿大学）；University of Amsterdam（阿姆斯特丹大学）；Umeå University（乌梅大学）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；TTIC（TTIC研究中心）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Theory of Mind and Self-Attributions of Mentality are Dissociable in LLMs
Google, Paradigms of Intelligence Team（Google，智能范式团队）；Knowledge Lab, University of Chicago（芝加哥大学知识实验室）；Institute of Philosophy, School of Advanced Study, University of London（伦敦大学高等研究院哲学研究所）；Department of Biomedical Informatics and Medical Education and Department of Bioethics and Humanities, School of Medicine, University of Washington（华盛顿大学医学院生物医学信息学与医学教育系及生物伦理学与人文学系）；Kellogg School of Management, Northwestern University（西北大学凯洛格管理学院）；Santa Fe Institute（圣塔菲研究所）
Measuring Representational Robustness of Neural Networks Through Shared Invariances
University of Maryland, College Park（马里兰大学帕克分校）；Max Planck Institute for Software Systems（马克斯·普朗克软件系统研究所）；PUCRS（巴西圣保罗天主教大学）；The Alan Turing Institute（艾伦·图灵研究所）；University of Cambridge（剑桥大学）
Measuring the Measures: Discriminative Capacity of Representational Similarity Metrics Across Model Families
Department of Computer Science and Engineering, UC San Diego（计算机科学与工程系，UC San Diego）；Department of Electrical and Computer Engineering, UC San Diego（电气与计算机工程系，UC San Diego）；Department of Cognitive Science, UC San Diego（认知科学系，UC San Diego）
Measuring Representation Robustness in Large Language Models for Geometry
Department of Computer Science and Information Systems, BITS Pilani（比特学院计算机科学与信息系统系）；School of Computer Science, KIIT University（KIIT大学计算机科学学院）；Department of Mathematics, BITS Pilani（比特学院数学系）
Tracing the Representation Geometry of Language Models from Pretraining to Post-training
Computer Science, McGill University（麦吉尔大学计算机科学系）；Mila - Quebec AI Institute（魁北克人工智能研究所）；UC Berkeley（加州大学伯克利分校）；G o o g l e , Paradigms of Intelligence Team（谷歌，智能范式团队）；Mathematics and Statistics, Université de Montréal（蒙特利尔大学数学与统计学系）；Neurology & Neurosurgery and Montreal Neurological Institute, McGill University（神经病学与神经外科及蒙特利尔神经研究所，麦吉尔大学）；CIFAR Learning in Machines & Brains Program（CIFAR 机器与大脑学习计划）
University of Technology Nuremberg（纽伦堡工业大学）；Carnegie Mellon University（卡内基梅隆大学）；International Institute of Information Technology, Hyderabad（海得拉巴国际信息技术学院）
Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)
Technische Universität Berlin（柏林技术大学）；IIIT Hyderabad（海得拉巴国家理工学院）；Univ of Maryland（马里兰大学）；Rice Univ（Rice 大学）；Univ of Wisconsin - Madison（威斯康星大学麦迪逊分校）；Spector Inc（Spector 公司）；Microsoft（微软公司）
