arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.09864cs.LGcs.AIcs.CLcs.CV
收藏

LUMA:一个用于从不确定和多模态数据中学习的基准数据集

LUMA: A Benchmark Dataset for Learning from Uncertain and Multimodal Data

Aix Marseille Univ, CNRS, LIS(阿维尼翁-马赛大学、国家科学研究中心、LIS)

Grigor Bezirganyan, Sana Sellami, Laure Berti-Équille, Sébastien Fournier

AI总结:

提出多模态数据集 LUMA,通过扩展 CIFAR 10/100 并引入音频与文本数据,支持受控注入不确定性,提供基线模型与三种不确定性量化方法,促进可信赖鲁棒多模态深度学习研究。

摘要:

多模态深度学习通过整合文本、图像、音频和视频等多样化信息源来增强决策能力。为了开发可信赖的多模态方法,理解不确定性如何影响这些模型至关重要。我们提出了 LUMA,一个独特的多模态数据集,包含来自 50 个类别的音频、图像和文本数据,专为从不确定数据中学习而设计。它通过提取自三个音频语料库的音频样本,以及使用 Gemma-7B 大语言模型(LLM)生成的文本数据,扩展了著名的 CIFAR 10/100 数据集。LUMA 数据集能够实现受控地注入不同类型和程度的不确定性,从而实现并定制特定的实验和基准测试计划。LUMA 也作为一个 Python 包提供,包含用于生成数据集多个变体的函数,可控制数据的多样性、每个模态的噪声量,并添加分布外样本。同时还提供了一个基线预训练模型以及三种不确定性量化方法:Monte-Carlo Dropout、Deep Ensemble 和 Reliable Conflictive Multi-View Learning。这个全面的数据集及其工具旨在促进和支持可信赖且鲁棒的多模态深度学习方法的开发、评估和基准测试。我们预计 LUMA 数据集将帮助研究社区为安全关键型应用设计更可信赖和鲁棒的机器学习方法。下载和处理数据集的代码及说明可在 https://github.com/bezirganyan/LUMA/ 找到。

英文摘要:

Multimodal Deep Learning enhances decision-making by integrating diverse information sources, such as texts, images, audio, and videos. To develop trustworthy multimodal approaches, it is essential to understand how uncertainty impacts these models. We propose LUMA, a unique multimodal dataset, featuring audio, image, and textual data from 50 classes, specifically designed for learning from uncertain data. It extends the well-known CIFAR 10/100 dataset with audio samples extracted from three audio corpora, and text data generated using the Gemma-7B Large Language Model (LLM). The LUMA dataset enables the controlled injection of varying types and degrees of uncertainty to achieve and tailor specific experiments and benchmarking initiatives. LUMA is also available as a Python package including the functions for generating multiple variants of the dataset with controlling the diversity of the data, the amount of noise for each modality, and adding out-of-distribution samples. A baseline pre-trained model is also provided alongside three uncertainty quantification methods: Monte-Carlo Dropout, Deep Ensemble, and Reliable Conflictive Multi-View Learning. This comprehensive dataset and its tools are intended to promote and support the development, evaluation, and benchmarking of trustworthy and robust multimodal deep learning approaches. We anticipate that the LUMA dataset will help the research community to design more trustworthy and robust machine learning approaches for safety critical applications. The code and instructions for downloading and processing the dataset can be found at: https://github.com/bezirganyan/LUMA/ .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑