多模态大模型

CM3AE: A Unified RGB Frame and Event-Voxel/-Frame Pre-training Framework

Wentao Wu, Xiao Wang, Chenglong Li, Bo Jiang, Jin Tang, Bin Luo, Qi Liu

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

From Trial to Triumph: Advancing Long Video Understanding via Visual Context Sample Scaling and Self-reward Alignment

Yucheng Suo, Fan Ma, Linchao Zhu, Tianyi Wang, Fengyun Rao, Yi Yang

专题命中 视频多模态 ：multi-modal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation

Kun Yuan, Vinkle Srivastav, Nassir Navab, Nicolas Padoy

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted at the 38th Conference on Neural Information Processing Systems (NeurIPS 2024 Spolight)

The Devil is in Temporal Token: High Quality Video Reasoning Segmentation

Sitong Gong, Yunzhi Zhuge, Lu Zhang, Zongxin Yang, Pingping Zhang, Huchuan Lu

专题命中 视频多模态 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Journal ref CVPR 2025

Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos

Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, Ziwei Liu

专题命中 视频多模态 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition

Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, Wynne Hsu

专题命中 视频多模态 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ICML 2024

Personality Analysis from Online Short Video Platforms with Multi-domain Adaptation

Sixu An, Xiangguo Sun, Yicong Li, Yu Yang, Guandong Xu

专题命中 视频多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Show and Guide: Instructional-Plan Grounded Vision and Language Model

Diogo Glória-Silva, David Semedo, João Magalhães

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted at EMNLP 2024 Main Track

Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering

Ting Yu, Kunhao Fu, Shuhui Wang, Qingming Huang, Jun Yu

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments IEEE Transactions on Circuits and Systems for Video Technology

Journal ref IEEE Transactions on Circuits and Systems for Video Technology, 2024

Encoding and Controlling Global Semantics for Long-form Video Question Answering

Thong Thanh Nguyen, Zhiyuan Hu, Xiaobao Wu, Cong-Duy T Nguyen, See-Kiong Ng, Anh Tuan Luu

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted to the main EMNLP 2024 conference

EAGLE: Egocentric AGgregated Language-video Engine

Jing Bi, Yunlong Tang, Luchuan Song, Ali Vosoughi, Nguyen Nguyen, Chenliang Xu

专题命中 视频多模态 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ACMMM 24

TC-LLaVA: Rethinking the Transfer from Image to Video Understanding with Temporal Considerations

Mingze Gao, Jingyu Liu, Mingda Li, Jiangtao Xie, Qingbin Liu, Bo Zhao, Xi Chen, Hui Xiong

专题命中 视频多模态 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos

Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, Hui Wang

专题命中 视频多模态 ：multi-modal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments 10 pages, 5 figures

Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization

Yang Jin, Zhicheng Sun, Kun Xu, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, Kun Gai, Yadong Mu

专题命中 视频多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Artemis: Towards Referential Understanding in Complex Videos

Jihao Qiu, Yuan Zhang, Xi Tang, Lingxi Xie, Tianren Ma, Pengyu Yan, David Doermann, Qixiang Ye, Yunjie Tian

专题命中 视频多模态 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments 19 pages, 14 figures. Code and data are available at https://github.com/qiujihao19/Artemis

Abductive Ego-View Accident Video Understanding for Safe Driving Perception

Jianwu Fang, Lei-lei Li, Junfei Zhou, Junbin Xiao, Hongkai Yu, Chen Lv, Jianru Xue, Tat-Seng Chua

专题命中 视频多模态 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by CVPR2024. This is not the camera-ready version. The Project page: http://www.lotvsmmau.net

SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks

Xingning Dong, Qingpei Guo, Tian Gan, Qing Wang, Jianlong Wu, Xiangyuan Ren, Yuan Cheng, Wei Chu

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted by TCSVT (IEEE Transactions on Circuits and Systems for Video Technology)

Conversation Understanding using Relational Temporal Graph Neural Networks with Auxiliary Cross-Modality Interaction

Cam-Van Thi Nguyen, Anh-Tuan Mai, The-Son Le, Hai-Dang Kieu, Duc-Trong Le

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.MM

Comments EMNLP 2023

Journal ref The 2023 Conference on Empirical Methods in Natural Language Processing

Learning Unseen Modality Interaction

Yunhua Zhang, Hazel Doughty, Cees G. M. Snoek

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Published at NeurIPS 2023

Improving Vision Anomaly Detection with the Guidance of Language Modality

Dong Chen, Kaihang Pan, Guoming Wang, Yueting Zhuang, Siliang Tang

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments 9 pages, 10 figures

ViGT: Proposal-free Video Grounding with Learnable Token in Transformer

Kun Li, Dan Guo, Meng Wang

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments This paper has been accepted by SCIENCE CHINA Information Sciences

$C^3$: Compositional Counterfactual Contrastive Learning for Video-grounded Dialogues

Hung Le, Nancy F. Chen, Steven C. H. Hoi

专题命中 视频多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.CL

Comments 24th Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)

Self-Supervised Learning for Videos: A Survey

Madeline C. Schiappa, Yogesh S. Rawat, Mubarak Shah

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments ACM CSUR (December 2022). Project Link: https://bit.ly/3Oimc7Q

Journal ref ACM Comput. Surv. (December 2022)

Contrastive Video Question Answering via Video Graph Transformer

Junbin Xiao, Pan Zhou, Angela Yao, Yicong Li, Richang Hong, Shuicheng Yan, Tat-Seng Chua

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted by IEEE T-PAMI'23

Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarks

Haiyang Xu, Qinghao Ye, Xuan Wu, Ming Yan, Yuan Miao, Jiabo Ye, Guohai Xu, Anwen Hu, Yaya Shi, Guangwei Xu, Chenliang Li, Qi Qian, Maofei Que, Ji Zhang, Xiao Zeng, Fei Huang

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Working in progress

Faster Video Moment Retrieval with Point-Level Supervision

Xun Jiang, Zailei Zhou, Xing Xu, Yang Yang, Guoqing Wang, Heng Tao Shen

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments 10 pages, 7 figures

HIINT: Historical, Intra- and Inter- personal Dynamics Modeling with Cross-person Memory Transformer

Yubin Kim, Dong Won Lee, Paul Pu Liang, Sharifa Algohwinem, Cynthia Breazeal, Hae Won Park

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Parallel Attention Network with Sequence Matching for Video Grounding

Hao Zhang, Aixin Sun, Wei Jing, Liangli Zhen, Joey Tianyi Zhou, Rick Siow Mong Goh

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments 15 pages, 10 figures, 7 tables, Findings at ACL 2021

Generation-Guided Multi-Level Unified Network for Video Grounding

Xing Cheng, Xiangyu Wu, Dong Shen, Hezheng Lin, Fan Yang

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferring

Ruyang Liu, Jingjia Huang, Ge Li, Jiashi Feng, Xinglong Wu, Thomas H. Li

专题命中 视频多模态 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI