Human-like object concept representations emerge naturally in multimodal large language models
State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology（脑认知与脑启发智能技术重点实验室）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Neuroscience, State Key Laboratory of Brain Cognition and Brain-Inspired Intelligence Technology（神经科学研究所）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）；Chinese Academy of Sciences（中国科学院）
Changde Du, Kaicheng Fu, Bincheng Wen, Yi Sun, Jie Peng, Wei Wei, Ying Gao, Shengpei Wang, Chuncheng Zhang, Jinpeng Li, Shuang Qiu, Le Chang, Huiguang He
理解人类如何对自然物体进行概念化与分类，能为感知和认知研究提供关键洞见。随着大语言模型（Large Language Models, LLMs）的出现，一个核心问题随之产生：这些模型能否从语言和多模态数据中发展出类人的物体表征？本研究结合行为分析与神经影像分析，探究LLM中的物体概念表征与人类认知之间的关联。我们从LLM及多模态大语言模型（Multimodal LLMs, MLLMs）中收集了470万组三元组判断数据，以此推导低维嵌入，捕捉1854个自然物体的相似性结构。最终得到的66维嵌入具备稳定性与预测性，且呈现出与人类心理表征相似的语义聚类特征。值得注意的是，这些嵌入背后的维度具备可解释性，表明LLM和MLLM发展出了类人的物体概念表征。进一步分析显示，模型嵌入与EBA、PPA、RSC、FFA等脑区的神经活动模式存在高度对齐。这提供了强有力的证据：尽管LLM中的物体表征与人类的并不完全相同，但二者存在根本相似性，反映了人类概念知识的关键特征。我们的研究结果推进了对机器智能的理解，并为开发更类人的人工认知系统提供了参考。
英文摘要：
Understanding how humans conceptualize and categorize natural objects offers critical insights into perception and cognition. With the advent of Large Language Models (LLMs), a key question arises: can these models develop human-like object representations from linguistic and multimodal data? In this study, we combined behavioral and neuroimaging analyses to explore the relationship between object concept representations in LLMs and human cognition. We collected 4.7 million triplet judgments from LLMs and Multimodal LLMs (MLLMs) to derive low-dimensional embeddings that capture the similarity structure of 1,854 natural objects. The resulting 66-dimensional embeddings were stable, predictive, and exhibited semantic clustering similar to human mental representations. Remarkably, the dimensions underlying these embeddings were interpretable, suggesting that LLMs and MLLMs develop human-like conceptual representations of objects. Further analysis showed strong alignment between model embeddings and neural activity patterns in brain regions such as EBA, PPA, RSC, and FFA. This provides compelling evidence that the object representations in LLMs, while not identical to human ones, share fundamental similarities that reflect key aspects of human conceptual knowledge. Our findings advance the understanding of machine intelligence and inform the development of more human-like artificial cognitive systems.
ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning
National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）；University of Washington（华盛顿大学）；Stanford University（斯坦福大学）；absolute AI；The Chinese University of Hong Kong（香港中文大学）
A Multimodal Large Language Model-Driven Framework for Context-Aware UAV Emergency Landing Site Selection
School of Information Science and Engineering, Southeast University（信息科学与工程学院，东南大学）；LASER, International Digital Economy Academy（LASER，国际数字经济学院）；Department of Electronic Engineering, East China Normal University（电子工程系，华东师范大学）
Currently, domestic service robots have an insufficient ability to interact naturally through language. This is because understanding human instructions is complicated by various ambiguities and missing information. In existing methods, the referring expressions that specify the relationships between objects are insufficiently modeled. In this paper, we propose Target-dependent UNITER, which learns the relationship between the target object and other objects directly by focusing on the relevant regions within an image, rather than the whole image. Our method is an extension of the UNITER-based…
Analyzing Diffusion and Autoregressive Vision Language Models in Multimodal Embedding Space
Nanyang Technological University（南洋理工大学）；Yale University（耶鲁大学）；NYU Shanghai（纽约大学上海分校）；Alibaba-NTU Singapore Joint Research Institute（阿里-国立新加坡大学联合研究机构）；University of the Chinese Academy of Sciences（中国科学院大学）；Center for Data Science（数据科学中心）；New York University（纽约大学）
Google Research（谷歌研究院）；Cornell University（康奈尔大学）；University of Copenhagen（哥本哈根大学）
Utilizing vision and language models (VLMs) pre-trained on large-scale image-text pairs is becoming a promising paradigm for open-vocabulary visual recognition. In this work, we extend this paradigm by leveraging motion and audio that naturally exist in video. We present \textbf{MOV}, a simple yet effective method for \textbf{M}ultimodal \textbf{O}pen-\textbf{V}ocabulary video classification. In MOV, we directly use the vision encoder from pre-trained VLMs with minimal modifications to encode video, optical flow and audio spectrogram. We design a cross-modal fusion mechanism to aggregate compl…
Montana State University（蒙大拿州立大学）；University of Colorado, Colorado Springs（科罗拉多大学科罗拉多斯普林斯分校）
Vision-language models can assess visual context in an image and generate descriptive text. While the generated text may be accurate and syntactically correct, it is often overly general. To address this, recent work has used optical character recognition to supplement visual information with text extracted from an image. In this work, we contend that vision-language models can benefit from additional information that can be extracted from an image, but are not used by current models. We modify previous multimodal frameworks to accept relevant information from any number of auxiliary classifie…
Knowledge-Augmented Multimodal Clinical Rationale Generation for Disease Diagnosis with Small Language Models
Hong Kong Baptist University（香港 Baptist 大学）；Shanxi University（山西大学）；Shanghai Institute for Advanced Study of Zhejiang University（浙江大学上海先进研究院）；Hong Kong University of Science and Technology（香港科技大学）；The University of Manchester（曼彻斯特大学）
Multimodal Large Language Models for Bioimage Analysis
National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理国家重点实验室）；Leibniz-Institut für Analytische Wissenschaften – ISAS – e.V.（莱布尼茨分析科学研究所）
Can Multimodal Large Language Model Think Analogically?
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Shandong Institutes of Industrial Technology（山东产业技术研究院）
Inria（法国国家信息与自动化研究所）；Université de Bordeaux（波尔多大学）；Microsoft Research（微软研究院）
In this paper, we investigate whether artificial agents can develop a shared language in an ecological setting where communication relies on a sensory-motor channel. To this end, we introduce the Graphical Referential Game (GREG) where a speaker must produce a graphical utterance to name a visual referent object while a listener has to select the corresponding object among distractor referents, given the delivered message. The utterances are drawing images produced using dynamical motor primitives combined with a sketching library. To tackle GREG we present CURVES: a multimodal contrastive dee…
EmergentBridge: Improving Zero-Shot Cross-Modal Transfer in Unified Multimodal Embedding Models
Tsinghua University（清华大学）；School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；JD iCity, JD Technology, JD Intelligent Cities Research（京东i城、京东科技、京东智能城市研究院）
Creating agents that can interact naturally with humans is a common goal in artificial intelligence (AI) research. However, evaluating these interactions is challenging: collecting online human-agent interactions is slow and expensive, yet faster proxy metrics often do not correlate well with interactive evaluation. In this paper, we assess the merits of these existing evaluation metrics and present a novel approach to evaluation called the Standardised Test Suite (STS). The STS uses behavioural scenarios mined from real human interaction data. Agents see replayed scenario context, receive an …
CACARA: Cross-Modal Alignment Leveraging a Text-Centric Approach for Cost-Effective Multimodal and Multilingual Learning
Instituto de Computação, Universidade Estadual de Campinas (UNICAMP), Brasil（计算机学院，Campinas州立大学（UNICAMP））；Instituto de Estudos da Linguagem, Universidade Estadual de Campinas (UNICAMP), Brasil（语言研究学院，Campinas州立大学（UNICAMP））；Instituto de Informática, Universidade Federal de Goiás (UFG), Goiás, Brasil（信息学院，戈亚斯联邦大学（UFG））；Department of Computer Science, University of Sheffield, Sheffield, United Kingdom（计算机科学系，谢菲尔德大学）
LVRPO: Language-Visual Alignment with GRPO for Multimodal Understanding and Generation
Department of Machine Learning, CMU, USA（卡内基梅隆大学机器学习系，美国）；Department of Machine Learning, MBZUAI, UAE（穆罕默德·本·扎耶德人工智能大学机器学习系，阿联酋）；Department of Artificial Intelligence, Hanyang University ERICA, South Korea（汉阳大学ERICA校区人工智能系，韩国）
Neural networks can be powerful function approximators, which are able to model high-dimensional feature distributions from a subset of examples drawn from the target distribution. Naturally, they perform well at generalizing within the limits of their target function, but they often fail to generalize outside of the explicitly learned feature space. It is therefore an open research topic whether and how neural network-based architectures can be deployed for systematic reasoning. Many studies have shown evidence for poor generalization, but they often work with abstract data or are limited to …
Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic Interactions
Graduate School of Artificial Intelligence, POSTECH（POSTECH人工智能研究生院）；Department of Computer Science and Engineering, POSTECH（POSTECH计算机科学与工程系）；Artificial Intelligence Graduate School, UNIST（UNIST人工智能研究生院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
IIT Kanpur（印度理工学院坎普尔分校）；University College London（伦敦大学学院）
A real-world application or setting involves interaction between different modalities (e.g., video, speech, text). In order to process the multimodal information automatically and use it for an end application, Multimodal Representation Learning (MRL) has emerged as an active area of research in recent times. MRL involves learning reliable and robust representations of information from heterogeneous sources and fusing them. However, in practice, the data acquired from different sources are typically noisy. In some extreme cases, a noise of large magnitude can completely alter the semantics of …
The wide application of smart devices enables the availability of multimodal data, which can be utilized in many tasks. In the field of multimodal sentiment analysis (MSA), most previous works focus on exploring intra- and inter-modal interactions. However, training a network with cross-modal information (language, visual, audio) is still challenging due to the modality gap, and existing methods still cannot ensure to sufficiently learn intra-/inter-modal dynamics. Besides, while learning dynamics within each sample draws great attention, the learning of inter-class relationships is neglected.…
