Recent Transformer-based contextual word representations, including BERT and XLNet, have shown state-of-the-art performance in multiple disciplines within NLP. Fine-tuning the trained contextual models on task-specific datasets has been the key to achieving superior performance downstream. While fine-tuning these pre-trained models is straightforward for lexical applications (applications with only language modality), it is not trivial for multimodal language (a growing area in NLP focused on modeling face-to-face communication). Pre-trained models don't have the necessary components to accept two extra modalities of vision and acoustic. In this paper, we proposed an attachment to BERT and XLNet called Multimodal Adaptation Gate (MAG). MAG allows BERT and XLNet to accept multimodal nonverbal data during fine-tuning. It does so by generating a shift to internal representation of BERT and XLNet; a shift that is conditioned on the visual and acoustic modalities. In our experiments, we study the commonly used CMU-MOSI and CMU-MOSEI datasets for multimodal sentiment analysis. Fine-tuning MAG-BERT and MAG-XLNet significantly boosts the sentiment analysis performance over previous baselines as well as language-only fine-tuning of BERT and XLNet. On the CMU-MOSI dataset, MAG-XLNet achieves human-level multimodal sentiment analysis performance for the first time in the NLP community.
This paper presents M3L-Contrast -- a novel multimodal multilingual (M3L) neural topic model for comparable data that maps texts from multiple languages and images into a shared topic space. Our model is trained jointly on texts and images and takes advantage of pretrained document and image embeddings to abstract the complexities between different languages and modalities. As a multilingual topic model, it produces aligned language-specific topics and as multimodal model, it infers textual representations of semantic concepts in images. We demonstrate that our model is competitive with a zero…
National Technical University of Athens（雅典国立技术大学）；Athena Research Center（雅典娜研究中心）
Multimodal learning pipelines have benefited from the success of pretrained language models. However, this comes at the cost of increased model parameters. In this work, we propose Adapted Multimodal BERT (AMB), a BERT-based architecture for multimodal tasks that uses a combination of adapter modules and intermediate fusion layers. The adapter adjusts the pretrained language model for the task at hand, while the fusion layers perform task-specific, layer-wise fusion of audio-visual information with textual BERT representations. During the adaptation process the pre-trained language model param…
Recent advances in learning aligned multimodal representations have been primarily driven by training large neural networks on massive, noisy paired-modality datasets. In this work, we ask whether it is possible to achieve similar results with substantially less training time and data. We achieve this by taking advantage of existing pretrained unimodal encoders and careful curation of alignment data relevant to the downstream task of interest. We study a natural approach to aligning existing encoders via small auxiliary functions, and we find that this method is competitive with (or outperform…
Beijing University of Posts and Telecommunications（北京邮电大学）
Existing multimodal conversation agents have shown impressive abilities to locate absolute positions or retrieve attributes in simple scenarios, but they fail to perform well when complex relative positions and information alignments are involved, which poses a bottleneck in response quality. In this paper, we propose a Situated Conversation Agent Petrained with Multimodal Questions from INcremental Layout Graph (SPRING) with abilities of reasoning multi-hops spatial relations and connecting them with visual attributes in crowded situated scenarios. Specifically, we design two types of Multimo…
Adjustable Text-Guided Backdoor Attacks with Natural-Word Triggers on Multimodal Pretrained Models
National Anti-Counterfeit Engineering Research Center, Huazhong University of Science and Technology（华中科技大学国家防伪工程技术研究中心）；School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；Sydney AI Centre, The University of Sydney（悉尼大学悉尼人工智能中心）
Shanghai Jiao Tong University（上海交通大学）；ProtagoLabs；Microsoft Research（微软研究院）；Chinese Academy of Sciences（中国科学院）；IGSNRR（中国科学院地理科学与资源研究所）
Relational structures such as schema linking and schema encoding have been validated as a key component to qualitatively translating natural language into SQL queries. However, introducing these structural relations comes with prices: they often result in a specialized model structure, which largely prohibits using large pretrained models in text-to-SQL. To address this problem, we propose RASAT: a Transformer seq2seq architecture augmented with relation-aware self-attention that could leverage a variety of relational structures while inheriting the pretrained parameters from the T5 model effe…
Pre-trained large language models have recently achieved ground-breaking performance in a wide variety of language understanding tasks. However, the same model can not be applied to multimodal behavior understanding tasks (e.g., video sentiment/humor detection) unless non-verbal features (e.g., acoustic and visual) can be integrated with language. Jointly modeling multiple modalities significantly increases the model complexity, and makes the training process data-hungry. While an enormous amount of text data is available via the web, collecting large-scale multimodal behavioral video datasets…
Columbia University（哥伦比亚大学）；Microsoft Research（微软研究院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Cross-modal encoders for vision-language (VL) tasks are often pretrained with carefully curated vision-language datasets. While these datasets reach an order of 10 million samples, the labor cost is prohibitive to scale further. Conversely, unimodal encoders are pretrained with simpler annotations that are less cost-prohibitive, achieving scales of hundreds of millions to billions. As a result, unimodal encoders have achieved state-of-art (SOTA) on many downstream tasks. However, challenges remain when applying to VL tasks. The pretraining data is not optimal for cross-modal architectures and …
FedNano: Toward Lightweight Federated Tuning for Pretrained Multimodal Large Language Models
LMU Munich（慕尼黑莱布尼茨大学）；Technical University of Munich（慕尼黑技术大学）；Siemens Technology（西门子技术）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University Heidelberg（海德堡大学）
National Taiwan University（台湾大学）；Academia Sinica（中央研究院）
Transformer-based models are widely used in natural language understanding (NLU) tasks, and multimodal transformers have been effective in visual-language tasks. This study explores distilling visual information from pretrained multimodal transformers to pretrained language encoders. Our framework is inspired by cross-modal encoders' success in visual-language tasks while we alter the learning objective to cater to the language-heavy characteristics of NLU. After training with a small number of extra adapting steps and finetuned, the proposed XDBERT (cross-modal distilled BERT) outperforms pre…
Word embeddings such as ELMo have recently been shown to model word semantics with greater efficacy through contextualized learning on large-scale language corpora, resulting in significant improvement in state of the art across many natural language tasks. In this work we integrate acoustic information into contextualized lexical embeddings through the addition of multimodal inputs to a pretrained bidirectional language model. The language model is trained on spoken language that includes text and audio modalities. The resulting representations from this model are multimodal and contain paral…
Multimodal-Multiresolution Foundation Model for Lunar Remote Sensing
IBM Research（IBM研究院）；The University of Alabama in Huntsville（阿拉巴马大学亨茨维尔分校）；University Space Research Association (USRA)（大学空间研究协会）；NASA Goddard Space Flight Center（美国国家航空航天局戈达德太空飞行中心）；SETI Institute（搜寻地外文明研究所）；NASA Ames Research Center（美国国家航空航天局艾姆斯研究中心）；University of Maryland, Baltimore County (UMBC)（马里兰大学巴尔的摩县分校）；Howard University（霍华德大学）
In today's increasingly international economy, return and volatility spillover effects across international equity markets are major macroeconomic drivers of stock dynamics. Thus, information regarding foreign markets is one of the most important factors in forecasting domestic stock prices. However, the cross-correlation between domestic and foreign markets is highly complex. Hence, it is extremely difficult to explicitly express this cross-correlation with a dynamical equation. In this study, we develop stock return prediction models that can jointly consider international markets, using mul…
Preliminary Investigations of a Multi-Faceted Robust and Synergistic Approach in Semiconductor Electron Micrograph Analysis: Integrating Vision Transformers with Large Language and Multimodal Models
ProMedTS: A Self-Supervised, Prompt-Guided Multimodal Approach for Integrating Medical Text and Time Series
Hong Kong Baptist University（香港 Baptist 大学）；Shanxi University（山西大学）；Shanghai Institute for Advanced Study of Zhejiang University（浙江大学上海研究院）；Tencent AI Lab（腾讯人工智能实验室）；Manchester Metropolitan University（曼彻斯特 Metropolitan 大学）；The University of Manchester（曼彻斯特大学）
Large pretrained (e.g., "foundation") models exhibit distinct capabilities depending on the domain of data they are trained on. While these domains are generic, they may only barely overlap. For example, visual-language models (VLMs) are trained on Internet-scale image captions, but large language models (LMs) are further trained on Internet-scale text with no images (e.g., spreadsheets, SAT questions, code). As a result, these models store different forms of commonsense knowledge across different domains. In this work, we show that this diversity is symbiotic, and can be leveraged through Soc…
Clinical decision-making in oncology involves multimodal data such as radiology scans, molecular profiling, histopathology slides, and clinical factors. Despite the importance of these modalities individually, no deep learning framework to date has combined them all to predict patient prognosis. Here, we predict the overall survival (OS) of glioma patients from diverse multimodal data with a Deep Orthogonal Fusion (DOF) model. The model learns to combine information from multiparametric MRI exams, biopsy-based modalities (such as H&E slide images and/or DNA sequencing), and clinical variables …
Enhancing Heavy Rain Nowcasting with Multimodal Data: Integrating Radar and Satellite Observations
Bochum Institute of Technology（波恩技术学院）；LMU Munich（慕尼黑大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Okeanos Smart Data Solutions GmbH（Okeanos智能数据解决方案有限公司）
Integrating Medical Imaging and Clinical Reports Using Multimodal Deep Learning for Advanced Disease Analysis
Penn State University（宾夕法尼亚州立大学）；Mississippi State University（密西西比州立大学）；Northwest Missouri State University（西北密苏里州立大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of California, Berkeley（加利福尼亚大学伯克利分校）；Colorado State University（科罗拉多州立大学）
Machine learning-based multimodal prognostic models integrating pathology images and high-throughput omic data for overall survival prediction in cancer: a systematic review
Leeds Teaching Hospitals NHS Trust（利兹教学医院国民医疗服务信托）；University of Leeds（利兹大学）；Linköping University（林雪平大学）
A Sparse-Attention Deep Learning Model Integrating Heterogeneous Multimodal Features for Parkinson's Disease Severity Profiling
School of Computing, Mathematics, and Engineering, Charles Sturt University（计算、数学与工程学院，查尔斯·斯特劳特大学）；School of Dentistry and Medical Sciences, Charles Sturt University（牙科学院与医学科学学院，查尔斯·斯特劳特大学）；AI and Cyber Futures Centre (AICF)（人工智能与未来技术中心（AICF））；Hawkins Clinic General Medical Practice（霍金斯诊所普通医疗实践）
Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding
InertialAI；Department of Electrical and Computer Engineering, Queen’s University（皇后大学电气与计算机工程系）；Department of Mechanical and Materials Engineering, Queen’s University（皇后大学机械与材料工程系）
In this paper, we address the problem of multimodal emotion recognition from multiple physiological signals. We demonstrate that a Transformer-based approach is suitable for this task. In addition, we present how such models may be pretrained in a multimodal scenario to improve emotion recognition performances. We evaluate the benefits of using multimodal inputs and pre-training with our approach on a state-ofthe-art dataset.
Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamics
Central Institute of Mental Health (CIMH), Medical Faculty Mannheim, Heidelberg University（海德堡大学曼heim医学院附属中央精神卫生研究所（CIMH））；Faculty of Physics and Astronomy, Heidelberg University（海德堡大学物理与天文学学院）；Interdisciplinary Center for Scientific Computing, Heidelberg University（海德堡大学跨学科科学计算中心）；Hector Institute for AI in Psychiatry, CIMH（中央精神卫生研究所赫克托精神病学人工智能研究所）
Longitudinal Multimodal Transformer Integrating Imaging and Latent Clinical Signatures From Routine EHRs for Pulmonary Nodule Classification
Vanderbilt University（范德堡大学）；Siemens Healthineers（西门子医疗）；Saint Luke’s Mid America Heart Institute（圣卢克中美洲心脏研究所）；Vanderbilt University Medical Center（范德堡大学医学中心）
Accurate and efficient product classification is significant for E-commerce applications, as it enables various downstream tasks such as recommendation, retrieval, and pricing. Items often contain textual and visual information, and utilizing both modalities usually outperforms classification utilizing either mode alone. In this paper we describe our methodology and results for the SIGIR eCom Rakuten Data Challenge. We employ a dual attention technique to model image-text relationships using pretrained language and image embeddings. While dual attention has been widely used for Visual Question…
Study of Emotion Concept Formation by Integrating Vision, Physiology, and Word Information using Multilayered Multimodal Latent Dirichlet Allocation
Graduate School of Engineering Science, Osaka University（大阪大学工程科学研究生院）；Graduate School of Science and Technology, Nara Institute of Science and Technology（奈良科学技术大学院大学科学技术研究生院）
