Multimodal Deep Learning enhances decision-making by integrating diverse information sources, such as texts, images, audio, and videos. To develop trustworthy multimodal approaches, it is essential to understand how uncertainty impacts these models. We propose LUMA, a unique multimodal dataset, featuring audio, image, and textual data from 50 classes, specifically designed for learning from uncertain data. It extends the well-known CIFAR 10/100 dataset with audio samples extracted from three audio corpora, and text data generated using the Gemma-7B Large Language Model (LLM). The LUMA dataset enables the controlled injection of varying types and degrees of uncertainty to achieve and tailor specific experiments and benchmarking initiatives. LUMA is also available as a Python package including the functions for generating multiple variants of the dataset with controlling the diversity of the data, the amount of noise for each modality, and adding out-of-distribution samples. A baseline pre-trained model is also provided alongside three uncertainty quantification methods: Monte-Carlo Dropout, Deep Ensemble, and Reliable Conflictive Multi-View Learning. This comprehensive dataset and its tools are intended to promote and support the development, evaluation, and benchmarking of trustworthy and robust multimodal deep learning approaches. We anticipate that the LUMA dataset will help the research community to design more trustworthy and robust machine learning approaches for safety critical applications. The code and instructions for downloading and processing the dataset can be found at: https://github.com/bezirganyan/LUMA/ .
LUMA-RAG: Lifelong Multimodal Agents with Provably Stable Streaming Alignment
Dept. of Computer Engineering(计算机工程系);SIES Graduate School of Technology(SIES技术研究生学院);Bharatiya Vidya Bhavan's Sardar Patel Institute of Technology(巴哈里亚·维达·巴万学院萨达尔·帕特尔技术学院)
MEWL: Few-shot multimodal word learning with referential uncertainty
Peking University(北京大学);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院);Beijing Institute of Technology(北京理工大学);Yangtze Delta Region Academy of Beijing Institute of Technology(北京理工大学长三角研究院)
MSEG-VCUQ: Multimodal SEGmentation with Enhanced Vision Foundation Models, Convolutional Neural Networks, and Uncertainty Quantification for High-Speed Video Phase Detection Data
Uncertainty-Aware Multimodal Learning via Conformal Shapley Intervals
Department of Statistics and Data Science, UCLA(统计与数据科学系, UCLA);Merck & Co(默克公司);Department of Biostatistics, UCLA(生物统计学系, UCLA);Departments of Statistics and Data Science, and of Biostatistics, UCLA(统计与数据科学系以及生物统计学系, UCLA)
Seeing and Reasoning with Confidence: Supercharging Multimodal LLMs with an Uncertainty-Aware Agentic Framework
University College London(伦敦大学学院);National Health Service(英国国家医疗服务体系);University of Surrey(萨里大学);Samsung AI Centre(三星人工智能中心);Queen Mary University of London(伦敦玛丽女王大学)
Uncertainty-Aware Multimodal Fusion for Oral Lesion Classification
1 School of Medical Science \& Technology, IIT Kharagpur, India 2 Department of Electrical Engineering, IIT Kharagpur, India 3 Dr. B.C. Roy Multispeciality Medical Research Centre, IIT Kharagpur, India
Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities
Academy for Engineering and Technology, Fudan University(复旦大学工程技术学院);Jilin Provincial Key Laboratory of Intelligence Science and Engineering(吉林省智能科学与工程重点实验室);Engineering Research Center of AI and Robotics, Ministry of Education(教育部人工智能与机器人工程研究中心)
Variational Neural Belief Parameterizations for Robust Dexterous Grasping under Multimodal Uncertainty
Department of Electrical & Computer Engineering and Institute for Systems Research, University of Maryland, College Park, MD, USA(电气与计算机工程系和系统研究所,马里兰大学,College Park, MD, USA);Maryland Applied Graduate Engineering, A. James Clark School of Engineering, University of Maryland, College Park, MD, USA(马里兰应用研究生工程学院,A. James Clark工程学院,马里兰大学,College Park, MD, USA)
FedUAF: Uncertainty-Aware Fusion with Reliability-Guided Aggregation for Multimodal Federated Sentiment Analysis
Shanghai University(上海大学);North China Electric Power University(华北电力大学);Université de Technologie de Compiègne(法国图卢兹国立理工学院);Nanyang Technological University(南洋理工大学);City University of Hong Kong(香港城市大学);Macquarie University(麦考瑞大学)
EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation
Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University(香港理工大学电子与电气工程系);Speech, Language, and Cognition Laboratory, The University of Hong Kong(香港大学语音、语言与认知实验室)