多模态大模型

StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identification

Yichen He, Yuan Lin, Jianchao Wu, Hanchong Zhang, Yuchen Zhang, Ruicheng Le

专题命中 视频多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Tempo-R0: A Video-MLLM for Temporal Video Grounding through Efficient Temporal Sensing Reinforcement Learning

Feng Yue, Zhaoxing Zhang, Junming Jiao, Zhengyu Liang, Shiwen Cao, Feifei Zhang, Rong Shen

专题命中 视频多模态 ：MLLM(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

M$^3$-Med: A Benchmark for Multi-lingual, Multi-modal, and Multi-hop Reasoning in Medical Instructional Video Understanding

Shenxi Liu, Kan Li, Mingyang Zhao, Yuhang Tian, Bin Li, Shoujun Zhou, Hongliang Li, Fuxia Yang

机构 * Beijing Institute of Technology（北京理工大学） ； The Hong Kong Polytechnic University（香港理工大学） ； Chinese Academy of Sciences（中国科学院） ； Air Force Hospital of Southern Theater Command of PLA（中国人民解放军南部战区空军医院） ； Shenzhen Traditional Chinese Medicine Hospital（深圳中医药医院） 纠错

专题命中 视频多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments 19 pages, 8 figures, 7 tables

Multimodal Alignment with Cross-Attentive GRUs for Fine-Grained Video Understanding

Namho Kim, Junhwa Kim

机构 * Korean Broadcasting System(KBS)（韩国广播系统） ； Department of Artificial Intelligence（人工智能系） ； Konyang University（全南大学） 纠错

专题命中 视频多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

COEF-VQ: Cost-Efficient Video Quality Understanding through a Cascaded Multimodal LLM Framework

Xin Dong, Sen Jia, Ming Rui Wang, Yan Li, Zhenheng Yang, Bingfeng Deng, Hongyu Xiong

机构 * ByteDance Inc.（字节跳动公司） 纠错

专题命中 视频多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representation

Yichen Xie, Runsheng Xu, Tong He, Jyh-Jing Hwang, Katie Luo, Jingwei Ji, Hubert Lin, Letian Chen, Yiren Lu, Zhaoqi Leng, Dragomir Anguelov, Mingxing Tan

机构 * UC Berkeley（加州大学伯克利分校） ； Waymo LLC（Waymo公司） ； Cornell University（康奈尔大学） ； Georgia Institute of Technology（佐治亚理工学院） 纠错

专题命中 视频多模态 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by CVPR2025; Project website: s4-driver.github.io

HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generation

Yicheng Xiao, Lin Song, Rui Yang, Cheng Cheng, Zunnan Xu, Zhaoyang Zhang, Yixiao Ge, Xiu Li, Ying Shan

机构 * Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学） ； ARC Lab, Tencent PCG（腾讯PCG ARC实验室） ； The University of Hong Kong（香港大学） ； Xi’an JiaoTong University（西安交通大学） 纠错

专题命中 视频多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

GODBench: A Benchmark for Multimodal Large Language Models in Video Comment Art

Yiming Lei, Chenkai Zhang, Zeming Liu, Haitao Leng, Shaoguo Liu, Tingting Gao, Qingjie Liu, Yunhong Wang

专题命中 视频多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL、cs.AI

Comments 69 pages, 66 figures, accepted by ACL 2025

TimeSoccer: An End-to-End Multimodal Large Language Model for Soccer Commentary Generation

Ling You, Wenxuan Huang, Xinni Xie, Xiangyi Wei, Bangyan Li, Shaohui Lin, Yang Li, Changbo Wang

机构 * East China Normal University（东华大学） 纠错

专题命中 视频多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL

MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA

Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang, Haoxuan You, Dan Xu, Zhe Gan, Jiasen Lu, Yinfei Yang

机构 * CSE, HKUST（香港科技大学计算机科学与工程学系） ； Apple（苹果公司） ； UCLA（加利福尼亚大学洛杉矶分校） 纠错

专题命中 视频多模态 ：multimodal(title,abstract)；multimodal foundation model(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ICLR 2025

Cross-modal Causal Relation Alignment for Video Question Grounding

Weixing Chen, Yang Liu, Binglin Chen, Jiandong Su, Yongsen Zheng, Liang Lin

机构 * Sun Yat-sen University（中山大学） ； Shenzhen Institute of Advanced Technology（深圳先进技术研究院） ； Nanyang Technological University（南洋理工大学） 纠错

专题命中 视频多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL

Comments Accepted by CVPR 2025

Multimodal Action Quality Assessment

Ling-An Zeng, Wei-Shi Zheng

机构 * School of Artificial Intelligence, Sun Yat-sen University（中山大学人工智能学院） ； School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院） 纠错

专题命中 视频多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments IEEE Transactions on Image Processing 2024

When Video Coding Meets Multimodal Large Language Models: A Unified Paradigm for Video Coding

Pingping Zhang, Jinlong Li, Kecheng Chen, Meng Wang, Long Xu, Haoliang Li, Nicu Sebe, Sam Kwong, Shiqi Wang

专题命中 视频多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.MM

Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models

Yifang Xu, Yunzhuo Sun, Benxiang Zhai, Ming Li, Wenxin Liang, Yang Li, Sidan Du

专题命中 视频多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.MM

Comments Accepted by AAAI 2025

H-MBA: Hierarchical MamBa Adaptation for Multi-Modal Video Understanding in Autonomous Driving

Siran Chen, Yuxiao Luo, Yue Ma, Yu Qiao, Yali Wang

专题命中 视频多模态 ：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments 7 pages, 4 figures

Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models

Jean Park, Kuk Jin Jang, Basam Alasaly, Sriharsha Mopidevi, Andrew Zolensky, Eric Eaton, Insup Lee, Kevin Johnson

专题命中 视频多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL、cs.AI

Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Experts

Chenyang Zhu, Bin Xiao, Lin Shi, Shoukun Xu, Xu Zheng

机构 * AI Thrust, HKUST(GZ)（香港科技大学（广州）AI部门） 纠错

专题命中 视频多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Dreaming User Multimodal Representation Guided by The Platonic Representation Hypothesis for Micro-Video Recommendation

Chengzhi Lin, Hezheng Lin, Shuchang Liu, Cangguang Ruan, LingJing Xu, Dezhao Yang, Chuyuan Wang, Yongqi Liu

专题命中 视频多模态 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Comments 4 Figure; 2 Table

LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Models

Qihang Ge, Wei Sun, Yu Zhang, Yunhao Li, Zhongpeng Ji, Fengyu Sun, Shangling Jui, Xiongkuo Min, Guangtao Zhai

专题命中 视频多模态 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

A Multimodal Transformer for Live Streaming Highlight Prediction

Jiaxin Deng, Shiyao Wang, Dong Shen, Liqin Zhao, Fan Yang, Guorui Zhou, Gaofeng Meng

专题命中 视频多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted at ICME 2024 as poster presentation. arXiv admin note: text overlap with arXiv:2306.14392

FreeVA: Offline MLLM as Training-Free Video Assistant

Wenhao Wu

专题命中 视频多模态 ：MLLM(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments Preprint. Work in progress

SpikeMba: Multi-Modal Spiking Saliency Mamba for Temporal Video Grounding

Wenrui Li, Xiaopeng Hong, Ruiqin Xiong, Xiaopeng Fan

专题命中 视频多模态 ：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM

VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning

Yanan Wang, Donghuo Zeng, Shinya Wada, Satoshi Kurihara

专题命中 视频多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.CL

Comments Accepted by IEEE Access

Journal ref in IEEE Access, vol. 11, pp. 51229-51240, 2023

Active Acquisition for Multimodal Temporal Data: A Challenging Decision-Making Task

Jannik Kossen, Cătălina Cangea, Eszter Vértes, Andrew Jaegle, Viorica Patraucean, Ira Ktena, Nenad Tomasev, Danielle Belgrave

专题命中 视频多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Published in Transactions on Machine Learning Research. Previous version accepted to Foundation Models for Decision Making Workshop at NeurIPS 2022

A Unified Multimodal De- and Re-coupling Framework for RGB-D Motion Recognition

Benjia Zhou, Pichao Wang, Jun Wan, Yanyan Liang, Fan Wang

专题命中 视频多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted to TPAMI 2023

MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition

Xinyu Gong, Sreyas Mohan, Naina Dhingra, Jean-Charles Bazin, Yilei Li, Zhangyang Wang, Rakesh Ranjan

专题命中 视频多模态 ：multi-modal(title)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted to CVPR 2023

Scalable and Accurate Self-supervised Multimodal Representation Learning without Aligned Video and Text Data

Vladislav Lialin, Stephen Rawls, David Chan, Shalini Ghosh, Anna Rumshisky, Wael Hamza

专题命中 视频多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Journal ref 2023 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW)

Frame-wise Cross-modal Matching for Video Moment Retrieval

Haoyu Tang, Jihua Zhu, Meng Liu, Zan Gao, Zhiyong Cheng

专题命中 视频多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM

Comments 12 pages; accepted by IEEE TMM

Journal ref IEEE Transactions on Multimedia 2021

Multimodal Event Transformer for Image-guided Story Ending Generation

Yucheng Zhou, Guodong Long

专题命中 视频多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments EACL 2023

Rethinking Multi-Modal Alignment in Video Question Answering from Feature and Sample Perspectives

Shaoning Xiao, Long Chen, Kaifeng Gao, Zhao Wang, Yi Yang, Zhimeng Zhang, Jun Xiao

专题命中 视频多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI