Determining the extent to which the perceptual world can be recovered from language is a longstanding problem in philosophy and cognitive science. We show that state-of-the-art large language models can unlock new insights into this problem by providing a lower bound on the amount of perceptual information that can be extracted from language. Specifically, we elicit pairwise similarity judgments from GPT models across six psychophysical datasets. We show that the judgments are significantly correlated with human data across all domains, recovering well-known representations like the color wheel and pitch spiral. Surprisingly, we find that a model (GPT-4) co-trained on vision and language does not necessarily lead to improvements specific to the visual modality. To study the influence of specific languages on perception, we also apply the models to a multilingual color-naming task. We find that GPT-4 replicates cross-linguistic variation in English and Russian illuminating the interaction of language and perception.
Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation
Columbia University（哥伦比亚大学）；University of Washington（华盛顿大学）
From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；School of Computer Science and Engineering, Tianjin University of Technology（天津理工大学计算机科学与工程学院）；Peng Cheng Laboratory（鹏城实验室）
Does Conceptual Representation Require Embodiment? Insights From Large Language Models
Basque Center on Cognition, Brain and Language（巴斯克认知、大脑与语言中心）；The Hong Kong Polytechnic University（香港理工大学）；Princeton Neuroscience Institute, Princeton University（普林斯顿大学普林斯顿神经科学研究所）；Hunter College, City University of New York（纽约城市大学亨特学院）；Graduate Center, City University of New York（纽约城市大学研究生中心）
QQJ: Quantifying Qualitative Judgment for Scalable and Human-Aligned Evaluation of Generative AI
AI Lab, Arioobarzan Engineering Team（艾伊罗巴赞工程团队人工智能实验室）；Department of Computer Engineering and Information Technology（计算机工程与信息科技系）；Department of Informatics, Bioengineering, Robotics and Systems Engineering（信息学、生物工程、机器人与系统工程系）；University of Genoa（热那亚大学）
A Survey of Generative Categories and Techniques in Multimodal Generative Models
School of Architecture, Technology and Engineering, University of Brighton, Lewes Road, BN2 4GJ（建筑、科技与工程学院，布里顿大学，勒斯路，BN2 4GJ）；University of Khorfakkan, Sharjah（柯法克坎大学，沙迦）
Words are all you need? Language as an approximation for human similarity judgments
Department of Psychology, Princeton University（普林斯顿大学心理学系）；Max Planck Institute for Empirical Aesthetics（马克斯·普朗克经验美学研究所）；Department of Computer Science, Princeton University（普林斯顿大学计算机科学系）；Max Planck Institute for Cognitive and Brain Sciences（马克斯·普朗克认知与脑科学研究所）
SonicBench: Dissecting the Physical Perception Bottleneck in Large Audio Language Models
Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative, Institute of Digital Twin, EIT（宁波空间智能与数字衍生关键实验室，数字孪生研究院，EIT）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究所，中国科学院）；Shenzhen University of Advanced Technology（深圳先进技术大学）；Shenzhen MSU-BIT University（深圳MSU-BIT大学）；Amazon AGI（亚马逊人工智能）
comp-syn: Perceptually Grounded Word Embeddings with Color
University of Chicago（芝加哥大学）；Psiphon Inc, Toronto（Psiphon公司（多伦多））；Stanford University（斯坦福大学）；University of California, Berkeley（加州大学伯克利分校）；Haas Business School（哈斯商学院）；Columbia University（哥伦比亚大学）；Sony CSL Paris（索尼CSL巴黎）
