多模态大模型 - arXivDaily 专题

2210.03625 2023-05-11 cs.CL cs.CV cs.MM 82%

C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval

Andrew Rouditchenko, Yung-Sung Chuang, Nina Shvetsova, Samuel Thomas, Rogerio Feris, Brian Kingsbury, Leonid Karlinsky, David Harwath, Hilde Kuehne, James Glass

专题命中视频多模态：cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted at ICASSP 2023. The code, models, and dataset are available at https://github.com/roudimit/c2kd

2305.03915 2023-05-09 cs.CV cs.CL cs.MM 82%

HateMM: A Multi-Modal Dataset for Hate Video Classification

Mithun Das, Rohit Raj, Punyajoy Saha, Binny Mathew, Manish Gupta, Animesh Mukherjee

专题命中视频多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted at ICWSM 2023(dataset track)

2208.08875 2023-04-12 physics.optics cs.GR 82%

Multi-modal imaging using a cascaded microscope design

Xi Yang, Mark Harfouche, Kevin C. Zhou, Lucas Kreiss, Shiqi Xu, Kanghyun Kim, Roarke Horstmeyer

专题命中视频多模态：multi-modal(title,abstract)；multimodal(abstract)

2207.01814 2023-01-23 cs.LG 82%

Multimodal Frame-Scoring Transformer for Video Summarization

Jeiyoon Park, Kiho Kwoun, Chanhee Lee, Heuiseok Lim

专题命中视频多模态：multimodal(title,abstract)；audio-visual(abstract)

Comments preprint

2210.08481 2022-10-18 cs.CV cs.CL cs.MM 82%

TLDW: Extreme Multimodal Summarisation of News Videos

Peggy Tang, Kun Hu, Lei Zhang, Jiebo Luo, Zhiyong Wang

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

2103.12541 2022-09-30 cs.MM cs.AI cs.CL cs.CR cs.CY cs.LG cs.SI 82%

A Survey on Multimodal Disinformation Detection

Firoj Alam, Stefano Cresci, Tanmoy Chakraborty, Fabrizio Silvestri, Dimiter Dimitrov, Giovanni Da San Martino, Shaden Shaar, Hamed Firooz, Preslav Nakov

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Accepted at COLING-2022, disinformation, misinformation, factuality, harmfulness, fake news, propaganda, multimodality, text, images, videos, network structure, temporality

2209.13537 2022-09-28 cs.NI cs.DB 82%

Sensing Multi-modal Mobility Patterns: A Case Study of Helsinki using Bluetooth Beacons and a Mobile Application

Zhiren Huang, Alonso Espinosa Mireles de Villafranca, Charalampos Sipetas

专题命中视频多模态：multi-modal(title,abstract)；multimodal(abstract)

2112.04446 2022-08-19 cs.CV cs.CL cs.SD eess.AS 82%

Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval

Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Rogerio Feris, David Harwath, James Glass, Hilde Kuehne

专题命中视频多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments CVPR2022. The final published version of the proceedings will be available on IEEE Xplore

2208.05814 2022-08-12 cs.CV cs.AI cs.MM 82%

Seeing your sleep stage: cross-modal distillation from EEG to infrared video

Jianan Han, Shaoxing Zhang, Aidong Men, Yang Liu, Ziming Yao, Yan Yan, Qingchao Chen

专题命中视频多模态：cross-modal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments We have submitted this paper to an academic journal

2206.07898 2022-06-17 cs.AI cs.CL cs.CV cs.LG 82%

Multimodal Dialogue State Tracking

Hung Le, Nancy F. Chen, Steven C. H. Hoi

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at NAACL 2022 (Oral)

2201.08264 2022-05-11 cs.CV cs.AI cs.CL cs.HC 82%

End-to-end Generative Pretraining for Multimodal Video Captioning

Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, Cordelia Schmid

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Journal ref Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) 2022

2204.03734 2022-04-11 cs.CV cs.CL cs.MM 82%

MHMS: Multimodal Hierarchical Multimedia Summarization

Jielin Qiu, Jiacheng Zhu, Mengdi Xu, Franck Dernoncourt, Trung Bui, Zhaowen Wang, Bo Li, Ding Zhao, Hailin Jin

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments 10 pages

2112.03763 2022-02-03 cs.LG 82%

Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning

DeepMind Interactive Agents Team, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico Carnevale, Mary Cassin, Felix Fischer, Petko Georgiev, Alex Goldin, Mansi Gupta, Tim Harley, Felix Hill, Peter C Humphreys, Alden Hung, Jessica Landon, Timothy Lillicrap, Hamza Merzic, Alistair Muldal, Adam Santoro, Guy Scully, Tamara von Glehn, Greg Wayne, Nathaniel Wong, Chen Yan, Rui Zhu

专题命中视频多模态：multimodal(title,abstract)；multi-modal(abstract)

2104.11178 2021-12-08 cs.CV cs.AI cs.LG cs.MM eess.IV 82%

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, Boqing Gong

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Published in the 35th Conference on Neural Information Processing Systems (NeurIPS 2021)

2010.11985 2021-04-30 cs.CL cs.CV cs.LG cs.MM 82%

MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences

Jianing Yang, Yongxin Wang, Ruitao Yi, Yuying Zhu, Azaan Rehman, Amir Zadeh, Soujanya Poria, Louis-Philippe Morency

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments NAACL 2021

2101.00848 2021-01-13 cs.CR 82%

Towards Cross-Modal Forgery Detection and Localization on Live Surveillance Videos

Yong Huang, Xiang Li, Wei Wang, Tao Jiang, Qian Zhang

专题命中视频多模态：cross-modal(title,abstract)；multi-modal(abstract)

Comments Accepted by IEEE International Conference on Computer Communications 2021

2002.06353 2020-09-16 cs.CV cs.CL cs.LG eess.AS eess.IV 82%

UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation

Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, Ming Zhou

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

2008.03546 2020-08-11 cs.CV cs.AI cs.LG cs.MM eess.IV 82%

Online Multi-modal Person Search in Videos

Jiangyue Xia, Anyi Rao, Qingqiu Huang, Linning Xu, Jiangtao Wen, Dahua Lin

专题命中视频多模态：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments ECCV2020. Project page: http://movienet.site/projects/eccv20onlineperson.html

2003.07758 2020-05-07 cs.CV cs.CL cs.LG cs.SD eess.AS eess.IV 82%

Multi-modal Dense Video Captioning

Vladimir Iashin, Esa Rahtu

专题命中视频多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments To appear in the proceedings of CVPR Workshops 2020; Code: https://github.com/v-iashin/MDVC Project Page: https://v-iashin.github.io/mdvc

2004.04917 2020-04-13 cs.LG cs.AI cs.CL cs.CV 82%

Multimodal Categorization of Crisis Events in Social Media

Mahdi Abavisani, Liwei Wu, Shengli Hu, Joel Tetreault, Alejandro Jaimes

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Conference on Computer Vision and Pattern Recognition (CVPR 2020)

Journal ref Conference on Computer Vision and Pattern Recognition (CVPR 2020)

1805.01060 2018-05-04 cs.AI cs.CL cs.CV 82%

Multimodal Emotion Recognition for One-Minute-Gradual Emotion Challenge

Ziqi Zheng, Chenjie Cao, Xingwei Chen, Guoqiang Xu

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

1712.01455 2017-12-06 cs.AI cs.CL cs.CV 82%

Multimodal Storytelling via Generative Adversarial Imitation Learning

Zhiqian Chen, Xuchao Zhang, Arnold P. Boedihardjo, Jing Dai, Chang-Tien Lu

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments IJCAI 2017

1701.03126 2017-03-13 cs.CV cs.CL cs.MM 82%

Attention-Based Multimodal Fusion for Video Description

Chiori Hori, Takaaki Hori, Teng-Yok Lee, Kazuhiro Sumi, John R. Hershey, Tim K. Marks

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Resubmitted to the rebuttal for CVPR 2017 for review, 8 pages, 4 figures

2603.11147 2026-04-03 cs.MM cs.CV cs.LG 82%

Catalogue Grounded Multimodal Attribution for Museum Video under Resource and Regulatory Constraints

博物馆视频下的资源与监管约束下的目录引导多模态归因

Minsak Nanang, Adrian Hilton, Armin Mustafa

机构 * University of Surrey（萨里大学）

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.MM

AI总结本文提出一种目录引导的多模态归因方法，用于博物馆视频内容的自动化元数据生成，以提高档案馆发现性并满足资源和监管要求。

Comments Demo video url: https://jn00767.pages.surrey.ac.uk/catalogue-grounded-multimodal-attribution-for-museum-video/

2601.21815 2026-02-02 cs.CY cs.AI cs.CL cs.SI 82%

Moral Outrage Shapes Commitments Beyond Attention: Multimodal Moral Emotions on YouTube in Korea and the US

道德愤怒影响承诺：韩国和美国YouTube上的多模态道德情感

Seongchan Park, Jaehong Kim, Hyeonseung Kim, Heejin Bin, Sue Moon, Wonjae Lee

机构 * KAIST（韩国科学技术院）

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CL、cs.AI

AI总结本研究通过多模态道德情感分类器分析YouTube上道德愤怒对用户参与度的影响，发现其在不同文化中均能提升观看和评论等互动行为。

Comments Accepted at The Web Conference 2026. We release Korean and English multimodal moral emotion classifiers

2502.12454 2025-08-13 cs.CV cs.AI cs.HC cs.LG 82%

Zero-shot Emotion Annotation in Facial Images Using Large Multimodal Models: Benchmarking and Prospects for Multi-Class, Multi-Frame Approaches

He Zhang, Xinyi Fu

机构 * Pennsylvania State University（宾夕法尼亚州立大学）； Tsinghua University（清华大学）

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments 10 pages, accepted to MRAC'25: 3rd International Workshop on Multimodal and Responsible Affective Computing (ACM-MM 2025)

2306.08657 2023-06-16 cs.CV cs.LG cs.MM 82%

EMERSK -- Explainable Multimodal Emotion Recognition with Situational Knowledge

Mijanur Palash, Bharat Bhargava

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.MM；multi-modal(comments)

Comments Emotion Recognition, Deep Learning, Multi-modal, Convolutional neural network (CNN), LSTM, Situational-Knowledge, Novelty

2005.13362 2020-05-29 cs.CL cs.CV 82%

A Multi-modal Approach to Fine-grained Opinion Mining on Video Reviews

Edison Marrese-Taylor, Cristian Rodriguez-Opazo, Jorge A. Balazs, Stephen Gould, Yutaka Matsuo

专题命中视频多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL；multimodal(comments)

Comments Second Grand Challenge and Workshop on Multimodal Language ACL 2020

2608.05780 2026-08-07 cs.CV 新提交 81%

Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding

用于高效长视频理解的证据驱动型动态视觉选择器

Bo Zhang, Wenxin Wang, Feng Chen, Zhihao Zhang, Zixuan Wang, Changsheng Li, Yinjie Lei

专题命中视频多模态：MLLM(summary_cn,abstract)；分类 cs.CV

AI总结本文提出基于目标MLLM内部注意力证据的动态视觉选择框架EviSelect，通过GRPO优化的随机策略实现高效长视频理解，在三个基准上性能更优，视觉token减少约50%、端到端加速3.9倍。

Comments Project Page: https://zhangbo135.github.io/EviSelect/

2602.03890 2026-06-05 cs.CV 81%

4DPC$^2$hat: Towards Dynamic Point Cloud Understanding with Failure-Aware Bootstrapping

4DPC$^2$hat: 面向动态点云理解的失败感知自举学习

Xindan Zhang, Weilong Yan, Yufei Shi, Xuerui Qiu, Tao He, Ying Li, Ming Li, Hehe Fan

机构 * University of Science and Technology of China（中国科学技术大学）

专题命中视频多模态：MLLM(abstract,abstract_cn)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

AI总结提出首个针对动态点云理解的多模态大语言模型4DPC$^2$hat，通过构建大规模跨模态数据集4DPC$^2$hat-200K和引入Mamba增强的时间推理模块及失败感知自举学习策略，显著提升了动作理解与时间推理能力。

Comments Accept by ICML 2026

1. 视频多模态 4735 篇

C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval

HateMM: A Multi-Modal Dataset for Hate Video Classification

Multi-modal imaging using a cascaded microscope design

Multimodal Frame-Scoring Transformer for Video Summarization

TLDW: Extreme Multimodal Summarisation of News Videos

A Survey on Multimodal Disinformation Detection

Sensing Multi-modal Mobility Patterns: A Case Study of Helsinki using Bluetooth Beacons and a Mobile Application

Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval

Seeing your sleep stage: cross-modal distillation from EEG to infrared video

Multimodal Dialogue State Tracking

End-to-end Generative Pretraining for Multimodal Video Captioning

MHMS: Multimodal Hierarchical Multimedia Summarization

Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences

Towards Cross-Modal Forgery Detection and Localization on Live Surveillance Videos

UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation

Online Multi-modal Person Search in Videos

Multi-modal Dense Video Captioning

Multimodal Categorization of Crisis Events in Social Media

Multimodal Emotion Recognition for One-Minute-Gradual Emotion Challenge

Multimodal Storytelling via Generative Adversarial Imitation Learning

Attention-Based Multimodal Fusion for Video Description

Catalogue Grounded Multimodal Attribution for Museum Video under Resource and Regulatory Constraints

Moral Outrage Shapes Commitments Beyond Attention: Multimodal Moral Emotions on YouTube in Korea and the US

Zero-shot Emotion Annotation in Facial Images Using Large Multimodal Models: Benchmarking and Prospects for Multi-Class, Multi-Frame Approaches

EMERSK -- Explainable Multimodal Emotion Recognition with Situational Knowledge

A Multi-modal Approach to Fine-grained Opinion Mining on Video Reviews

Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding

4DPC$^2$hat: Towards Dynamic Point Cloud Understanding with Failure-Aware Bootstrapping