多模态大模型 - arXivDaily 专题

2407.08150 2024-09-06 cs.CV 83%

Hypergraph Multi-modal Large Language Model: Exploiting EEG and Eye-tracking Modalities to Evaluate Heterogeneous Responses for Video Understanding

Minghui Wu, Chenxu Zhao, Anyang Su, Donglin Di, Tianyu Fu, Da An, Min He, Ya Gao, Meng Ma, Kun Yan, Ping Wang

专题命中视频多模态：multi-modal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments Accepted by ACM MULTIMEDIA 2024

2408.01766 2024-08-20 cs.CV 83%

MultiFuser: Multimodal Fusion Transformer for Enhanced Driver Action Recognition

Ruoyu Wang, Wenqian Wang, Jianjun Gao, Dan Lin, Kim-Hui Yap, Bingbing Li

专题命中视频多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2408.05523 2024-08-15 cs.HC cs.CV 83%

DeepFace-Attention: Multimodal Face Biometrics for Attention Estimation with Application to e-Learning

Roberto Daza, Luis F. Gomez, Julian Fierrez, Aythami Morales, Ruben Tolosana, Javier Ortega-Garcia

专题命中视频多模态：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Article accepted in the IEEE Access journal. Accessible at https://ieeexplore.ieee.org/document/10633208

2406.11340 2024-08-06 cs.CV cs.LG 83%

CM2-Net: Continual Cross-Modal Mapping Network for Driver Action Recognition

Ruoyu Wang, Chen Cai, Wenqian Wang, Jianjun Gao, Dan Lin, Wenyang Liu, Kim-Hui Yap

专题命中视频多模态：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

2407.09157 2024-07-15 cs.IR cs.AI cs.LG 83%

Movie Recommendation with Poster Attention via Multi-modal Transformer Feature Fusion

Linhan Xia, Yicheng Yang, Ziou Chen, Zheng Yang, Shengxin Zhu

专题命中视频多模态：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.AI

2405.19654 2024-05-31 cs.AI 83%

Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training

Jinxia Yang, Bing Su, Wayne Xin Zhao, Ji-Rong Wen

专题命中视频多模态：multimodal(title)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.AI

Comments Accepted at ICML 2024

2307.03638 2024-05-21 cs.MM cs.HC 83%

Physical-aware Cross-modal Adversarial Network for Wearable Sensor-based Human Action Recognition

Jianyuan Ni, Hao Tang, Anne H. H. Ngu, Gaowen Liu, Yan Yan

专题命中视频多模态：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.MM

Comments We will be making some significant changes to the paper, including the title and methodology. We therefore wish to withdraw the paper for now

2404.09010 2024-04-16 cs.CV cs.LG 83%

MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild

Kateryna Chumachenko, Alexandros Iosifidis, Moncef Gabbouj

专题命中视频多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments accepted to CVPR 2024 ABAW Workshop

2404.08949 2024-04-16 cs.CL 83%

Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles

Abhijnan Nath, Huma Jamil, Shafiuddin Rehan Ahmed, George Baker, Rahul Ghosh, James H. Martin, Nathaniel Blanchard, Nikhil Krishnaswamy

专题命中视频多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL

Comments To appear at LREC-COLING 2024

2301.07868 2024-04-12 cs.CV 83%

MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval

Xiaojie Jin, Bowen Zhang, Weibo Gong, Kai Xu, XueQing Deng, Peng Wang, Zhao Zhang, Xiaohui Shen, Jiashi Feng

专题命中视频多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

2404.03413 2024-04-05 cs.CV 83%

MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens

Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, Mohamed Elhoseiny

专题命中视频多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments 6 pages,8 figures

2402.19258 2024-03-01 cs.CV 83%

MaskFi: Unsupervised Learning of WiFi and Vision Representations for Multimodal Human Activity Recognition

Jianfei Yang, Shijie Tang, Yuecong Xu, Yunjiao Zhou, Lihua Xie

专题命中视频多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments 9 pages

2212.00773 2024-02-09 cs.CV 83%

FakeOut: Leveraging Out-of-domain Self-supervision for Multi-modal Video Deepfake Detection

Gil Knafo, Ohad Fried

专题命中视频多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV

2401.17797 2024-02-01 cs.CV 83%

M2-RAAP: A Multi-Modal Recipe for Advancing Adaptation-based Pre-training towards Effective and Efficient Zero-shot Video-text Retrieval

Xingning Dong, Zipeng Feng, Chunluan Zhou, Xuzheng Yu, Ming Yang, Qingpei Guo

专题命中视频多模态：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV

2401.01674 2024-01-04 cs.CV 83%

Transformer RGBT Tracking with Spatio-Temporal Multimodal Tokens

Dengdi Sun, Yajie Pan, Andong Lu, Chenglong Li, Bin Luo

专题命中视频多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2312.12721 2023-12-21 cs.CV 83%

Cross-Modal Reasoning with Event Correlation for Video Question Answering

Chengxiang Yin, Zhengping Che, Kun Wu, Zhiyuan Xu, Qinru Qiu, Jian Tang

专题命中视频多模态：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

2304.08083 2023-12-14 cs.CV 83%

VCD: Visual Causality Discovery for Cross-Modal Question Reasoning

Yang Liu, Ying Tan, Jingzhou Luo, Weixing Chen

专题命中视频多模态：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments 12 pages, 6 figures. arXiv admin note: substantial text overlap with arXiv:2207.12647

2309.15915 2023-09-29 cs.CV 83%

Zero-Shot and Few-Shot Video Question Answering with Multi-Modal Prompts

Deniz Engin, Yannis Avrithis

专题命中视频多模态：multi-modal(title)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments ICCV2023 CLVL Workshop (Oral). Project page: https://engindeniz.github.io/vitis

2309.15313 2023-09-28 cs.CV 83%

M$^{3}$3D: Learning 3D priors using Multi-Modal Masked Autoencoders for 2D image and video understanding

Muhammad Abdullah Jamal, Omid Mohareri

专题命中视频多模态：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2309.15082 2023-09-27 cs.CV 83%

RPEFlow: Multimodal Fusion of RGB-PointCloud-Event for Joint Optical Flow and Scene Flow Estimation

Zhexiong Wan, Yuxin Mao, Jing Zhang, Yuchao Dai

专题命中视频多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments ICCV 2023. Project page: https://npucvr.github.io/RPEFlow Code: https://github.com/danqu130/RPEFlow

2309.02041 2023-09-06 cs.CV 83%

Learning Cross-Modal Affinity for Referring Video Object Segmentation Targeting Limited Samples

Guanghui Li, Mingqi Gao, Heng Liu, Xiantong Zhen, Feng Zheng

专题命中视频多模态：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments Accepted by ICCV2023

2209.12244 2023-08-24 cs.CV 83%

Multimodal Channel-Mixing: Channel and Spatial Masked AutoEncoder on Facial Action Unit Detection

Xiang Zhang, Huiyuan Yang, Taoyue Wang, Xiaotian Li, Lijun Yin

专题命中视频多模态：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

2308.09775 2023-08-22 cs.CV 83%

Long-range Multimodal Pretraining for Movie Understanding

Dawit Mureja Argaw, Joon-Young Lee, Markus Woodson, In So Kweon, Fabian Caba Heilbron

专题命中视频多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted to ICCV 2023

2305.14652 2023-06-01 cs.CL 83%

Denoising Bottleneck with Mutual Information Maximization for Video Multimodal Fusion

Shaoxiang Wu, Damai Dai, Ziwei Qin, Tianyu Liu, Binghuai Lin, Yunbo Cao, Zhifang Sui

专题命中视频多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CL

Comments Accept at ACL2023

2210.10842 2023-05-09 cs.CV cs.LG cs.RO 83%

MMRNet: Improving Reliability for Multimodal Object Detection and Segmentation for Bin Picking via Multimodal Redundancy

Yuhao Chen, Hayden Gunraj, E. Zhixuan Zeng, Robbie Meyer, Maximilian Gilles, Alexander Wong

专题命中视频多模态：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted to CVPR TCV Workshop

2209.05419 2023-05-01 cs.CV 83%

Multimodal Graph Learning for Deepfake Detection

Zhiyuan Yan, Peng Sun, Yubo Lang, Shuo Du, Shanzhuo Zhang, Wei Wang, Lei Liu

专题命中视频多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2303.17285 2023-03-31 cs.CV 83%

Decomposed Cross-modal Distillation for RGB-based Temporal Action Detection

Pilhyeon Lee, Taeoh Kim, Minho Shim, Dongyoon Wee, Hyeran Byun

专题命中视频多模态：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments Accepted by CVPR 2023

2211.14905 2023-03-28 cs.CV cs.AI cs.CL cs.LG cs.MM 83%

Multi-Modal Few-Shot Temporal Action Detection

Sauradip Nag, Mengmeng Xu, Xiatian Zhu, Juan-Manuel Perez-Rua, Bernard Ghanem, Yi-Zhe Song, Tao Xiang

专题命中视频多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Technical Report

2303.10826 2023-03-28 cs.CV 83%

Visual Prompt Multi-Modal Tracking

Jiawen Zhu, Simiao Lai, Xin Chen, Dong Wang, Huchuan Lu

专题命中视频多模态：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments Accepted by CVPR2023

2303.00040 2023-03-27 cs.CV 83%

Towards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-Text Pre-Training

Dezhao Luo, Jiabo Huang, Shaogang Gong, Hailin Jin, Yang Liu

专题命中视频多模态：image-text(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments CVPR2023

1. 视频多模态 4729 篇

Hypergraph Multi-modal Large Language Model: Exploiting EEG and Eye-tracking Modalities to Evaluate Heterogeneous Responses for Video Understanding

MultiFuser: Multimodal Fusion Transformer for Enhanced Driver Action Recognition

DeepFace-Attention: Multimodal Face Biometrics for Attention Estimation with Application to e-Learning

CM2-Net: Continual Cross-Modal Mapping Network for Driver Action Recognition

Movie Recommendation with Poster Attention via Multi-modal Transformer Feature Fusion

Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training

Physical-aware Cross-modal Adversarial Network for Wearable Sensor-based Human Action Recognition

MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild

Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles

MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval

MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens

MaskFi: Unsupervised Learning of WiFi and Vision Representations for Multimodal Human Activity Recognition

FakeOut: Leveraging Out-of-domain Self-supervision for Multi-modal Video Deepfake Detection

M2-RAAP: A Multi-Modal Recipe for Advancing Adaptation-based Pre-training towards Effective and Efficient Zero-shot Video-text Retrieval

Transformer RGBT Tracking with Spatio-Temporal Multimodal Tokens

Cross-Modal Reasoning with Event Correlation for Video Question Answering

VCD: Visual Causality Discovery for Cross-Modal Question Reasoning

Zero-Shot and Few-Shot Video Question Answering with Multi-Modal Prompts

M$^{3}$3D: Learning 3D priors using Multi-Modal Masked Autoencoders for 2D image and video understanding

RPEFlow: Multimodal Fusion of RGB-PointCloud-Event for Joint Optical Flow and Scene Flow Estimation

Learning Cross-Modal Affinity for Referring Video Object Segmentation Targeting Limited Samples

Multimodal Channel-Mixing: Channel and Spatial Masked AutoEncoder on Facial Action Unit Detection

Long-range Multimodal Pretraining for Movie Understanding

Denoising Bottleneck with Mutual Information Maximization for Video Multimodal Fusion

MMRNet: Improving Reliability for Multimodal Object Detection and Segmentation for Bin Picking via Multimodal Redundancy

Multimodal Graph Learning for Deepfake Detection

Decomposed Cross-modal Distillation for RGB-based Temporal Action Detection

Multi-Modal Few-Shot Temporal Action Detection

Visual Prompt Multi-Modal Tracking

Towards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-Text Pre-Training