The task of video and text sequence alignment is a prerequisite step toward joint understanding of movie videos and screenplays. However, supervised methods face the obstacle of limited realistic training data. With this paper, we attempt to enhance data efficiency of the end-to-end alignment network NeuMATCH [15]. Recent research [56] suggests that network components dealing with different modalities may overfit and generalize at different speeds, creating difficulties for training. We propose to employ (1) layer-wise adaptive rate scaling (LARS) to align the magnitudes of gradient updates in different layers and balance the pace of learning and (2) sequence-wise batch normalization (SBN) to align the internal feature distributions from different modalities. Finally, we leverage random projection to reduce the dimensionality of input features. On the YouTube Movie Summary dataset, the combined use of these technique closes the performance gap when the pretraining on the LSMDC dataset is omitted and achieves the state-of-the-art result. Extensive empirical comparisons and analysis reveal that these techniques improve optimization and regularize the network more effectively than two different setups of layer normalization.
Aligning First, Then Fusing: A Novel Weakly Supervised Multimodal Violence Detection Method
School of Software Engineering（软件工程学院）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Xi’an Jiaotong University（西安交通大学）；Chinese Academy of Sciences（中国科学院）
A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence. In this work, we introduce Kosmos-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot). Specifically, we train Kosmos-1 from scratch on web-scale multimodal corpora, including arbitrarily interleaved text and images, image-caption pairs, and text data. We evaluate various settings, including zero-shot, few-shot, and multimodal chain-of-thought prompting, on…
International Institute of Information Technology Hyderabad（海得拉巴国际信息技术学院）
Multimodal Machine Translation (MMT) enriches the source text with visual information for translation. It has gained popularity in recent years, and several pipelines have been proposed in the same direction. Yet, the task lacks quality datasets to illustrate the contribution of visual modality in the translation systems. In this paper, we propose our system under the team name Volta for the Multimodal Translation Task of WAT 2021 from English to Hindi. We also participate in the textual-only subtask of the same language pair for which we use mBART, a pretrained multilingual sequence-to-sequen…
Aligning Multimodal LLM with Human Preference: A Survey
Institute of automation, Chinese academy of science（中国科学院自动化研究所）；Nanjing University（南京大学）；University of Science and Technology of China（中国科学技术大学）；Nanyang Technological University（南洋理工大学）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Xiamen University（厦门大学）；National University of Singapore（新加坡国立大学）；Lehigh University（理海大学）；The Hong Kong University of Science and Technology（香港科技大学）；Squirrel Ai Learning（松鼠AI学习）
One of the key challenges in learning joint embeddings of multiple modalities, e.g. of images and text, is to ensure coherent cross-modal semantics that generalize across datasets. We propose to address this through joint Gaussian regularization of the latent representations. Building on Wasserstein autoencoders (WAEs) to encode the input in each domain, we enforce the latent embeddings to be similar to a Gaussian prior that is shared across the two domains, ensuring compatible continuity of the encoded semantic representations of images and texts. Semantic alignment is achieved through superv…
Aligning by Misaligning: Boundary-aware Curriculum Learning for Multimodal Alignment
Nanjing University（南京大学）；Airon Technology CO., LTD（艾润科技有限公司）；Shanghai Jiao Tong University（上海交通大学）；University of Bristol（布里斯托大学）；The Hong Kong Polytechnic University（香港理工大学）；The University of Hong Kong（香港大学）；Carnegie Mellon University（卡内基梅隆大学）
Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning
Tongji University（同济大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；East China Normal University（华东师范大学）；Stanford University（斯坦福大学）；New York University（纽约大学）
本文提出 VaLiK 方法，通过跨模态信息补充构建无需注释的多模态知识图谱，提升大语言模型推理能力。
Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Large Language Model Department, Tencent（腾讯大语言模型部）；University of Electronic Science and Technology of China（电子科技大学）；Hong Kong University of Science and Technology（香港科技大学）；Zhongguancun Academy（中关村学院）
Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentration
University of Technology Sydney（悉尼科技大学）；East China Normal University（华东师范大学）；The University of New South Wales（新南威尔士大学）；Shanghai Jiaotong University（上海交通大学）；The Education University of Hong Kong（香港教育大学）
Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所信息与智能系统研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Zhejiang University（浙江大学）
Polytechnique Montreal（蒙特利尔理工学院）；Universite de Montreal（蒙特利尔大学）；MILA（蒙特利尔学习算法研究所）
The goal of vision-language modeling is to allow models to tie language understanding with visual inputs. The aim of this paper is to evaluate and align the Visual Language Model (VLM) called Multimodal Augmentation of Generative Models through Adapter-based finetuning (MAGMA) with human values. MAGMA is a VLM that is capable of image captioning and visual question-answering. We will evaluate its alignment in three different scenarios. To begin, we assess MAGMA's out-of-the-box alignment through the checkpoint provided by Hugging Face. Then, we measure if few-shot learning manages to improve t…
Toward Unified Multimodal Representation Learning for Autonomous Driving
J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA（德克萨斯大学机械工程系，德克萨斯农工大学，学院站，德克萨斯，77843，美国）；The Department of Engineering Technology and Industrial Distribution Texas A&M University, College Station, TX 77843, USA（工程技术与工业分布系，德克萨斯农工大学，学院站，德克萨斯，77843，美国）
ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Pengcheng Laboratory（鹏城实验室）；University of Oxford（牛津大学）；Oxford-Suzhou Centre for Advanced Research（牛津-苏州高等研究院）；Tencent AI Lab（腾讯AI实验室）
Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion
ZIP Lab, Zhejiang University（浙江大学 ZIP 实验室）；South China University of Technology（华南理工大学）；Pazhou Lab（琶洲实验室）；Key Laboratory of Big Data and Intelligent Robot, Ministry of Education（大数据与智能机器人教育部重点实验室）
MoECa: Aligning Feature Reuse with Expert Decomposition in Diffusion Transformers
School of Computer Science, Peking University（北京大学计算机科学学院）；School of Software Engineering, University of Electronic Science and Technology of China（电子科技大学软件工程学院）
The Wellcome Centre for Human Neuroimaging, UCL（伦敦大学学院惠康人类神经影像中心）
In medical imaging it is common practice to acquire a wide range of modalities (MRI, CT, PET, etc.), to highlight different structures or pathologies. As patient movement between scans or scanning session is unavoidable, registration is often an essential step before any subsequent image analysis. In this paper, we introduce a cost function based on joint total variation for such multimodal image registration. This cost function has the advantage of enabling principled, groupwise alignment of multiple images, whilst being insensitive to strong intensity non-uniformities. We evaluate our algori…
StructXLIP: Enhancing Vision-language Models with Multimodal Structural Cues
University of Verona（维罗纳大学）；Sapienza University of Rome（罗马大学）；University of Trento（特伦托大学）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）；Reykjavik University（雷克雅未克大学）
mChartQA: A universal benchmark for multimodal Chart Question Answer based on Vision-Language Alignment and Reasoning
Shenyang Institute of Computing Technology, Chinese Academy of Sciences（中国科学院沈阳计算技术研究所）；Beijing Wenge Technology Co., Ltd（北京文格科技有限公司）；University of Chinese Academy of Sciences（中国科学院大学）
Visual-Oriented Fine-Grained Knowledge Editing for MultiModal Large Language Models
Hefei University of Technology（合肥工业大学）；University of Science and Technology of China（中国科学技术大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）
A Practitioner's Guide to Continual Multimodal Pretraining
Tübingen AI Center（蒂宾根人工智能中心）；University of Tübingen（蒂宾根大学）；Helmholtz Munich（亥姆霍兹慕尼黑中心）；University of Cambridge（剑桥大学）；LAION；Jülich Supercomputing Center (JSC)（于利希超级计算中心）；Research Center Jülich (FZJ)（于利希研究中心）；Helmholtz Association（亥姆霍兹联合会）；Google DeepMind（谷歌DeepMind）；Munich Center for ML（慕尼黑机器学习中心）；Technical University of Munich（慕尼黑工业大学）
MaViLS, a Benchmark Dataset for Video-to-Slide Alignment, Assessing Baseline Accuracy with a Multimodal Alignment Algorithm Leveraging Speech, OCR, and Visual Features
Karlsruhe University of Applied Sciences（卡尔斯鲁厄应用技术大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；University of Hohenheim（霍恩海姆大学）
