Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning
Tongji University（同济大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；East China Normal University（华东师范大学）；Stanford University（斯坦福大学）；New York University（纽约大学）
Junming Liu, Siyuan Meng, Yanting Gao, Song Mao, Pinlong Cai, Guohang Yan, Yirong Chen, Zilin Bian, Ding Wang, Botian Shi
备注：
14 pages, 7 figures, 6 tables; Accepted by ICCV 2025
Multimodal reasoning in Large Language Models (LLMs) struggles with incomplete knowledge and hallucination artifacts, challenges that textual Knowledge Graphs (KGs) only partially mitigate due to their modality isolation. While Multimodal Knowledge Graphs (MMKGs) promise enhanced cross-modal understanding, their practical construction is impeded by semantic narrowness of manual text annotations and inherent noise in visual-semantic entity linkages. In this paper, we propose Vision-align-to-Language integrated Knowledge Graph (VaLiK), a novel approach for constructing MMKGs that enhances LLMs reasoning through cross-modal information supplementation. Specifically, we cascade pre-trained Vision-Language Models (VLMs) to align image features with text, transforming them into descriptions that encapsulate image-specific information. Furthermore, we developed a cross-modal similarity verification mechanism to quantify semantic consistency, effectively filtering out noise introduced during feature alignment. Even without manually annotated image captions, the refined descriptions alone suffice to construct the MMKG. Compared to conventional MMKGs construction paradigms, our approach achieves substantial storage efficiency gains while maintaining direct entity-to-image linkage capability. Experimental results on multimodal reasoning tasks demonstrate that LLMs augmented with VaLiK outperform previous state-of-the-art models. Our code is published at https://github.com/Wings-Of-Disaster/VaLiK.
Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentration
University of Technology Sydney（悉尼科技大学）；East China Normal University（华东师范大学）；The University of New South Wales（新南威尔士大学）；Shanghai Jiaotong University（上海交通大学）；The Education University of Hong Kong（香港教育大学）
TV-SAM: Increasing Zero-Shot Segmentation Performance on Multimodal Medical Images Using GPT-4 Generated Descriptive Prompts Without Human Annotation
Sichuan University（四川大学）；Case Western Reserve University（凯斯西储大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Urgench State University（乌尔根奇国立大学）；Gansu Agricultural University（甘肃农业大学）
Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion
ZIP Lab, Zhejiang University（浙江大学 ZIP 实验室）；South China University of Technology（华南理工大学）；Pazhou Lab（琶洲实验室）；Key Laboratory of Big Data and Intelligent Robot, Ministry of Education（大数据与智能机器人教育部重点实验室）
MELT: Towards Automated Multimodal Emotion Data Annotation by Leveraging LLM Embedded Knowledge
Technical University of Munich（慕尼黑技术大学）；Imperial College London（伦敦帝国理工学院）；CHI – Chair of Health Informatics（健康信息学系）；Munich Centre for Machine Learning（慕尼黑机器学习中心）；GLAM – Group on Language, Audio, & Music（语言、音频与音乐小组）
Are Multimodal Large Language Models Good Annotators for Image Tagging?
RIKEN Center for Advanced Intelligence Project, Japan（日本先进人工智能研究中心）；The University of Tokyo, Japan（东京大学）；Southeast University, China（东南大学）；China University of Mining（中国矿业大学）
Large Language Models Meet Harry Potter: A Bilingual Dataset for Aligning Dialogue Agents with Characters
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；Tencent AI Lab（腾讯人工智能实验室）
Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Large Language Model Department, Tencent（腾讯大语言模型部）；University of Electronic Science and Technology of China（电子科技大学）；Hong Kong University of Science and Technology（香港科技大学）；Zhongguancun Academy（中关村学院）
Aligning Large Language Models and Geometric Deep Models for Protein Representation
Northwestern University（西北大学）；University of Pittsburgh（匹兹堡大学）；Michigan State University（密歇根州立大学）；North Carolina State University（北卡罗来纳州立大学）；New Jersey Institute of Technology（新泽西理工学院）
Aligning by Misaligning: Boundary-aware Curriculum Learning for Multimodal Alignment
Nanjing University（南京大学）；Airon Technology CO., LTD（艾润科技有限公司）；Shanghai Jiao Tong University（上海交通大学）；University of Bristol（布里斯托大学）；The Hong Kong Polytechnic University（香港理工大学）；The University of Hong Kong（香港大学）；Carnegie Mellon University（卡内基梅隆大学）
Aligning First, Then Fusing: A Novel Weakly Supervised Multimodal Violence Detection Method
School of Software Engineering（软件工程学院）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Xi’an Jiaotong University（西安交通大学）；Chinese Academy of Sciences（中国科学院）
Data-efficient Alignment of Multimodal Sequences by Aligning Gradient Updates and Internal Feature Distributions
Fudan University（复旦大学）；Nanyang Technological University（南洋理工大学）；Alibaba-NTU Singapore Joint Research Institute（阿里巴巴-南洋理工大学新加坡联合研究院）
视频和文本序列对齐的任务是共同理解电影视频和电影剧本的先决条件。然而，监督方法面临着现实训练数据有限的障碍。在本文中，我们试图提高端到端对齐网络NeuMATCH的数据效率[15]。最近的研究[56]表明，处理不同模态的网络组件可能会以不同的速度过拟合和泛化，从而给训练带来困难。我们建议采用（1）逐层自适应速率缩放（LARS）来对齐不同层中梯度更新的幅度并平衡学习的速度，以及（2）逐序列批量归一化（SBN）来对齐来自不同模态的内部特征分布。最后，我们利用随机投影来降低输入特征的维数。在YouTube Movie Summary数据集上，当省略LSMDC数据集上的预训练时，这些技术的组合使用缩小了性能差距，并实现了最先进的结果。广泛的实证比较和分析表明，这些技术提高了优化和正规化网络更有效地比两个不同的设置层归一化。
