多模态大模型 - arXivDaily 专题

2210.08481 2022-10-18 cs.CV cs.CL cs.MM 82%

TLDW: Extreme Multimodal Summarisation of News Videos

Peggy Tang, Kun Hu, Lei Zhang, Jiebo Luo, Zhiyong Wang

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

2103.12541 2022-09-30 cs.MM cs.AI cs.CL cs.CR cs.CY cs.LG cs.SI 82%

A Survey on Multimodal Disinformation Detection

Firoj Alam, Stefano Cresci, Tanmoy Chakraborty, Fabrizio Silvestri, Dimiter Dimitrov, Giovanni Da San Martino, Shaden Shaar, Hamed Firooz, Preslav Nakov

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Accepted at COLING-2022, disinformation, misinformation, factuality, harmfulness, fake news, propaganda, multimodality, text, images, videos, network structure, temporality

2209.13537 2022-09-28 cs.NI cs.DB 82%

Sensing Multi-modal Mobility Patterns: A Case Study of Helsinki using Bluetooth Beacons and a Mobile Application

Zhiren Huang, Alonso Espinosa Mireles de Villafranca, Charalampos Sipetas

专题命中视频多模态：multi-modal(title,abstract)；multimodal(abstract)

2112.04446 2022-08-19 cs.CV cs.CL cs.SD eess.AS 82%

Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval

Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Rogerio Feris, David Harwath, James Glass, Hilde Kuehne

专题命中视频多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments CVPR2022. The final published version of the proceedings will be available on IEEE Xplore

2208.05814 2022-08-12 cs.CV cs.AI cs.MM 82%

Seeing your sleep stage: cross-modal distillation from EEG to infrared video

Jianan Han, Shaoxing Zhang, Aidong Men, Yang Liu, Ziming Yao, Yan Yan, Qingchao Chen

专题命中视频多模态：cross-modal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments We have submitted this paper to an academic journal

2206.07898 2022-06-17 cs.AI cs.CL cs.CV cs.LG 82%

Multimodal Dialogue State Tracking

Hung Le, Nancy F. Chen, Steven C. H. Hoi

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at NAACL 2022 (Oral)

2201.08264 2022-05-11 cs.CV cs.AI cs.CL cs.HC 82%

End-to-end Generative Pretraining for Multimodal Video Captioning

Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, Cordelia Schmid

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Journal ref Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) 2022

2204.03734 2022-04-11 cs.CV cs.CL cs.MM 82%

MHMS: Multimodal Hierarchical Multimedia Summarization

Jielin Qiu, Jiacheng Zhu, Mengdi Xu, Franck Dernoncourt, Trung Bui, Zhaowen Wang, Bo Li, Ding Zhao, Hailin Jin

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments 10 pages

2112.03763 2022-02-03 cs.LG 82%

Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning

DeepMind Interactive Agents Team, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico Carnevale, Mary Cassin, Felix Fischer, Petko Georgiev, Alex Goldin, Mansi Gupta, Tim Harley, Felix Hill, Peter C Humphreys, Alden Hung, Jessica Landon, Timothy Lillicrap, Hamza Merzic, Alistair Muldal, Adam Santoro, Guy Scully, Tamara von Glehn, Greg Wayne, Nathaniel Wong, Chen Yan, Rui Zhu

专题命中视频多模态：multimodal(title,abstract)；multi-modal(abstract)

2104.11178 2021-12-08 cs.CV cs.AI cs.LG cs.MM eess.IV 82%

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, Boqing Gong

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Published in the 35th Conference on Neural Information Processing Systems (NeurIPS 2021)

2010.11985 2021-04-30 cs.CL cs.CV cs.LG cs.MM 82%

MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences

Jianing Yang, Yongxin Wang, Ruitao Yi, Yuying Zhu, Azaan Rehman, Amir Zadeh, Soujanya Poria, Louis-Philippe Morency

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments NAACL 2021

2101.00848 2021-01-13 cs.CR 82%

Towards Cross-Modal Forgery Detection and Localization on Live Surveillance Videos

Yong Huang, Xiang Li, Wei Wang, Tao Jiang, Qian Zhang

专题命中视频多模态：cross-modal(title,abstract)；multi-modal(abstract)

Comments Accepted by IEEE International Conference on Computer Communications 2021

2002.06353 2020-09-16 cs.CV cs.CL cs.LG eess.AS eess.IV 82%

UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation

Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, Ming Zhou

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

2008.03546 2020-08-11 cs.CV cs.AI cs.LG cs.MM eess.IV 82%

Online Multi-modal Person Search in Videos

Jiangyue Xia, Anyi Rao, Qingqiu Huang, Linning Xu, Jiangtao Wen, Dahua Lin

专题命中视频多模态：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments ECCV2020. Project page: http://movienet.site/projects/eccv20onlineperson.html

2003.07758 2020-05-07 cs.CV cs.CL cs.LG cs.SD eess.AS eess.IV 82%

Multi-modal Dense Video Captioning

Vladimir Iashin, Esa Rahtu

专题命中视频多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments To appear in the proceedings of CVPR Workshops 2020; Code: https://github.com/v-iashin/MDVC Project Page: https://v-iashin.github.io/mdvc

2004.04917 2020-04-13 cs.LG cs.AI cs.CL cs.CV 82%

Multimodal Categorization of Crisis Events in Social Media

Mahdi Abavisani, Liwei Wu, Shengli Hu, Joel Tetreault, Alejandro Jaimes

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Conference on Computer Vision and Pattern Recognition (CVPR 2020)

Journal ref Conference on Computer Vision and Pattern Recognition (CVPR 2020)

1805.01060 2018-05-04 cs.AI cs.CL cs.CV 82%

Multimodal Emotion Recognition for One-Minute-Gradual Emotion Challenge

Ziqi Zheng, Chenjie Cao, Xingwei Chen, Guoqiang Xu

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

1712.01455 2017-12-06 cs.AI cs.CL cs.CV 82%

Multimodal Storytelling via Generative Adversarial Imitation Learning

Zhiqian Chen, Xuchao Zhang, Arnold P. Boedihardjo, Jing Dai, Chang-Tien Lu

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments IJCAI 2017

1701.03126 2017-03-13 cs.CV cs.CL cs.MM 82%

Attention-Based Multimodal Fusion for Video Description

Chiori Hori, Takaaki Hori, Teng-Yok Lee, Kazuhiro Sumi, John R. Hershey, Tim K. Marks

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Resubmitted to the rebuttal for CVPR 2017 for review, 8 pages, 4 figures

2603.11147 2026-04-03 cs.MM cs.CV cs.LG 82%

Catalogue Grounded Multimodal Attribution for Museum Video under Resource and Regulatory Constraints

博物馆视频下的资源与监管约束下的目录引导多模态归因

Minsak Nanang, Adrian Hilton, Armin Mustafa

机构 * University of Surrey（萨里大学）

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.MM

AI总结本文提出一种目录引导的多模态归因方法，用于博物馆视频内容的自动化元数据生成，以提高档案馆发现性并满足资源和监管要求。

Comments Demo video url: https://jn00767.pages.surrey.ac.uk/catalogue-grounded-multimodal-attribution-for-museum-video/

2601.21815 2026-02-02 cs.CY cs.AI cs.CL cs.SI 82%

Moral Outrage Shapes Commitments Beyond Attention: Multimodal Moral Emotions on YouTube in Korea and the US

道德愤怒影响承诺：韩国和美国YouTube上的多模态道德情感

Seongchan Park, Jaehong Kim, Hyeonseung Kim, Heejin Bin, Sue Moon, Wonjae Lee

机构 * KAIST（韩国科学技术院）

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CL、cs.AI

AI总结本研究通过多模态道德情感分类器分析YouTube上道德愤怒对用户参与度的影响，发现其在不同文化中均能提升观看和评论等互动行为。

Comments Accepted at The Web Conference 2026. We release Korean and English multimodal moral emotion classifiers

2502.12454 2025-08-13 cs.CV cs.AI cs.HC cs.LG 82%

Zero-shot Emotion Annotation in Facial Images Using Large Multimodal Models: Benchmarking and Prospects for Multi-Class, Multi-Frame Approaches

He Zhang, Xinyi Fu

机构 * Pennsylvania State University（宾夕法尼亚州立大学）； Tsinghua University（清华大学）

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments 10 pages, accepted to MRAC'25: 3rd International Workshop on Multimodal and Responsible Affective Computing (ACM-MM 2025)

2306.08657 2023-06-16 cs.CV cs.LG cs.MM 82%

EMERSK -- Explainable Multimodal Emotion Recognition with Situational Knowledge

Mijanur Palash, Bharat Bhargava

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.MM；multi-modal(comments)

Comments Emotion Recognition, Deep Learning, Multi-modal, Convolutional neural network (CNN), LSTM, Situational-Knowledge, Novelty

2005.13362 2020-05-29 cs.CL cs.CV 82%

A Multi-modal Approach to Fine-grained Opinion Mining on Video Reviews

Edison Marrese-Taylor, Cristian Rodriguez-Opazo, Jorge A. Balazs, Stephen Gould, Yutaka Matsuo

专题命中视频多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL；multimodal(comments)

Comments Second Grand Challenge and Workshop on Multimodal Language ACL 2020

2608.05780 2026-08-07 cs.CV 新提交 81%

Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding

用于高效长视频理解的证据驱动型动态视觉选择器

Bo Zhang, Wenxin Wang, Feng Chen, Zhihao Zhang, Zixuan Wang, Changsheng Li, Yinjie Lei

专题命中视频多模态：MLLM(summary_cn,abstract)；分类 cs.CV

AI总结本文提出基于目标MLLM内部注意力证据的动态视觉选择框架EviSelect，通过GRPO优化的随机策略实现高效长视频理解，在三个基准上性能更优，视觉token减少约50%、端到端加速3.9倍。

Comments Project Page: https://zhangbo135.github.io/EviSelect/

2602.03890 2026-06-05 cs.CV 81%

4DPC$^2$hat: Towards Dynamic Point Cloud Understanding with Failure-Aware Bootstrapping

4DPC$^2$hat: 面向动态点云理解的失败感知自举学习

Xindan Zhang, Weilong Yan, Yufei Shi, Xuerui Qiu, Tao He, Ying Li, Ming Li, Hehe Fan

机构 * University of Science and Technology of China（中国科学技术大学）

专题命中视频多模态：MLLM(abstract,abstract_cn)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

AI总结提出首个针对动态点云理解的多模态大语言模型4DPC$^2$hat，通过构建大规模跨模态数据集4DPC$^2$hat-200K和引入Mamba增强的时间推理模块及失败感知自举学习策略，显著提升了动作理解与时间推理能力。

Comments Accept by ICML 2026

2008.13369 2020-10-27 cs.CV cs.HC cs.LG 81%

Introducing Representations of Facial Affect in Automated Multimodal Deception Detection

Leena Mathur, Maja J Matarić

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV

Comments 10 pages, Accepted at ACM International Conference on Multimodal Interaction (ICMI), October 2020

Journal ref Proceedings of the 2020 International Conference on Multimodal Interaction (ICMI)

2608.28383 2026-08-31 cs.CV cs.CL 新提交 81%

Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs

语义头专业化指导多模态大语言模型的混合视觉Transformer注意力机制

Chenhong He, Lei Li, Shicheng Li, Hanglong Lv, Lingpeng Kong, Qi Liu, Tong Yang, Shuhuai Ren

机构 * Peking University（北京大学）； Xiaomi Corporation（小米公司）； The University of Hong Kong（香港大学）

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL

AI总结该研究针对多模态LLM中混合ViT注意力设计不足的问题，提出语义头专业化（SHS）概念，据此设计Ariadne注意力机制，在22项图像视频任务上性能与全注意力相当，计算量降低6.5倍。

2608.22617 2026-08-25 cs.CV cs.AI cs.HC 新提交 81%

AI-based worker guidance in assembly and disassembly operations using multimodal ego/exo-centric data capture and structured task knowledge

基于多模态自我/外部中心数据采集与结构化任务知识的装配与拆卸作业中基于人工智能的工人指导

Vivek Chavan, Jörg Krüger

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI

AI总结该研究提出一种从专家演示中提取结构化任务知识的以数据为中心的方法，结合多模态数据推导任务表示，实现装配拆卸作业的工人指导，经案例验证其优于静态图像方法，可用于维修、培训等场景。

Comments 5 pages. Published in CIRP Annals - Manufacturing Technology

Journal ref CIRP Annals - Manufacturing Technology 75 (2026) 19-23

2608.21030 2026-08-24 cs.CV cs.CL cs.LG 新提交 81%

COMET: Contrastive Motion-Enhanced Temporal Reasoning for Video Multimodal Large Language Models

COMET：面向视频多模态大语言模型的对比运动增强时序推理

Chenghua Zhu, Zhaolu Kang, Qifan Shi, Siyan Wu, Kehan Jiang, Lei Wei, Lianyu Hu, Guangyuan Dong, Mingbo Yang, Rui Lu, Guibo Luo

机构 * Peking University（北京大学）； Nanyang Technological University（南洋理工大学）； National University of Singapore（新加坡国立大学）； Sun Yat-Sen University（中山大学）； Pingan Technology（平安科技）

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL

AI总结 COMET是一种视频多模态大语言模型时序增强框架，通过显式时序表示等方法，在Qwen3-VL-8B等模型的动作、时序推理任务上取得性能提升，且可跨模型家族通用。

Comments Accepted at the 34th ACM International Conference on Multimedia (ACM MM 2026)

1. 视频多模态 4778 篇

TLDW: Extreme Multimodal Summarisation of News Videos

A Survey on Multimodal Disinformation Detection

Sensing Multi-modal Mobility Patterns: A Case Study of Helsinki using Bluetooth Beacons and a Mobile Application

Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval

Seeing your sleep stage: cross-modal distillation from EEG to infrared video

Multimodal Dialogue State Tracking

End-to-end Generative Pretraining for Multimodal Video Captioning

MHMS: Multimodal Hierarchical Multimedia Summarization

Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences

Towards Cross-Modal Forgery Detection and Localization on Live Surveillance Videos

UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation

Online Multi-modal Person Search in Videos

Multi-modal Dense Video Captioning

Multimodal Categorization of Crisis Events in Social Media

Multimodal Emotion Recognition for One-Minute-Gradual Emotion Challenge

Multimodal Storytelling via Generative Adversarial Imitation Learning

Attention-Based Multimodal Fusion for Video Description

Catalogue Grounded Multimodal Attribution for Museum Video under Resource and Regulatory Constraints

Moral Outrage Shapes Commitments Beyond Attention: Multimodal Moral Emotions on YouTube in Korea and the US

Zero-shot Emotion Annotation in Facial Images Using Large Multimodal Models: Benchmarking and Prospects for Multi-Class, Multi-Frame Approaches

EMERSK -- Explainable Multimodal Emotion Recognition with Situational Knowledge

A Multi-modal Approach to Fine-grained Opinion Mining on Video Reviews

Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding

4DPC$^2$hat: Towards Dynamic Point Cloud Understanding with Failure-Aware Bootstrapping

Introducing Representations of Facial Affect in Automated Multimodal Deception Detection

Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs

AI-based worker guidance in assembly and disassembly operations using multimodal ego/exo-centric data capture and structured task knowledge

COMET: Contrastive Motion-Enhanced Temporal Reasoning for Video Multimodal Large Language Models