传统的大型多模态模型(LMMs)在视频音频理解方面通常仅在几分钟长的短视频上进行评估。本文介绍了QMAVIS(Q Team-Multimodal Audio Video Intelligent Sensemaking),一种通过后期融合LMMs、大型语言模型和语音识别模型构建的新型长视频音频理解流程。QMAVIS填补了长视频分析的空白,特别是针对几分钟到超过一小时长的视频,为感知、视频内容分析、具身AI等开辟了新的应用前景。使用QMAVIS进行定量实验表明,在包含音频信息的VideoMME数据集上,其性能比最先进的视频音频LMMs如VideoLlaMA2和InternVL2提高了38.75%。在其他具有挑战性的视频理解数据集如PerceptionTest和EgoSchema上,最高达到了2%的提升,显示出竞争力。定性实验也显示,QMAVIS能够提取长视频音频内容中不同场景的细微差别并理解整体叙事。此外,还进行了消融研究以确定融合流程中每个组件的影响。
英文摘要:
Large Multimodal Models (LMMs) for video-audio understanding have traditionally been evaluated only on shorter videos of a few minutes long. In this paper, we introduce QMAVIS (Q Team-Multimodal Audio Video Intelligent Sensemaking), a novel long video-audio understanding pipeline built through a late fusion of LMMs, Large Language Models, and speech recognition models. QMAVIS addresses the gap in long-form video analytics, particularly for longer videos of a few minutes to beyond an hour long, opening up new potential applications in sensemaking, video content analysis, embodied AI, etc. Quantitative experiments using QMAVIS demonstrated a 38.75% improvement over state-of-the-art video-audio LMMs like VideoLlaMA2 and InternVL2 on the VideoMME (with subtitles) dataset, which comprises long videos with audio information. Evaluations on other challenging video understanding datasets like PerceptionTest and EgoSchema saw up to 2% improvement, indicating competitive performance. Qualitative experiments also showed that QMAVIS is able to extract the nuances of different scenes in a long video audio content while understanding the overarching narrative. Ablation studies were also conducted to ascertain the impact of each component in the fusion pipeline.
Evaluating Hallucinations in Audio-Visual Multimodal LLMs with Spoken Queries under Diverse Acoustic Conditions
Department of Intelligent Software, Sungkyunkwan University(智能软件系,成均馆大学);Department of Computer Science and Engineering, Sungkyunkwan University(计算机科学与工程系,成均馆大学)
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
Columbia University(哥伦比亚大学);Google(谷歌);Cornell University(康奈尔大学)
我们提出了一个框架,学习多模态表示从未标记的数据使用卷积自由Transformer架构。具体来说,我们的视频-音频-文本Transformer(VATT)将原始信号作为输入,并提取多模态表示,这些表示足够丰富,可以使各种下游任务受益。我们使用多模态对比损失从头开始训练VATT端到端,并通过视频动作识别、音频事件分类、图像分类和文本到视频检索等下游任务来评估其性能。此外,我们研究了一个模态不可知的,单骨干Transformer共享三个模态之间的权重。我们证明了无卷积VATT在下游任务中的性能优于最先进的基于ConvNet的架构。特别是,VATT的视觉Transformer在Kinetics-400上达到了82.1%的顶级准确率,在Kinetics-600上达到了83.6%,在Kinetics-700上达到了72.7%,在Moments in Time上达到了41.1%,这些都是新的记录,同时避免了监督式的预训练。转移到图像分类导致ImageNet上的top-1准确率为78.7%,而从头开始训练相同的Transformer的准确率为64.7%,这表明尽管视频和图像之间存在领域差距,但我们的模型具有普遍性。VATT的音频Transformer还在基于波形的音频事件识别方面创造了新纪录,在AudioSet上实现了39.4%的mAP,而无需任何监督的预训练。VATT的源代码是公开的。
Enriching Multimodal Sentiment Analysis through Textual Emotional Descriptions of Visual-Audio Content
Tianjin University(天津大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室(深圳));Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
Multimodal Audio-based Disease Prediction with Transformer-based Hierarchical Fusion Network
Purdue University(普渡大学);Beijing University of Chemical Technology(北京化工大学);New York University(纽约大学);University of Cincinnati(辛辛那提大学);Indiana University(印第安纳大学)