多模态大模型 - arXivDaily 专题

2412.18748 2025-01-03 cs.MM cs.CL cs.SD eess.AS 82%

Towards Expressive Video Dubbing with Multiscale Multimodal Context Interaction

Yuan Zhao, Rui Liu, Gaoxiang Cong

机构 * Inner Mongolia University（内蒙古大学）； Inst. of Comput. Tech., CAS（中国科学院计算技术研究所）

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted by ICASSP 2025

2410.16407 2024-10-23 cs.CL cs.AI cs.MM 82%

Enhancing Multimodal Affective Analysis with Learned Live Comment Features

Zhaoyuan Deng, Amith Ananthram, Kathleen McKeown

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

2410.10818 2024-10-16 cs.CV cs.AI cs.CL cs.LG 82%

TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models

Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, Yao Dou, Jaden Park, Jianfeng Gao, Yong Jae Lee, Jianwei Yang

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project Page: https://temporalbench.github.io/

2402.19467 2024-10-11 cs.CL cs.AI cs.CV 82%

TV-TREES: Multimodal Entailment Trees for Neuro-Symbolic Video Reasoning

Kate Sanders, Nathaniel Weir, Benjamin Van Durme

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 9 pages, EMNLP 2024

2405.15766 2024-10-04 cs.AI cs.CL cs.CV 82%

Enhancing Adverse Drug Event Detection with Multimodal Dataset: Corpus Creation and Model Development

Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Aman Chadha, Samrat Mondal

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL Findings 2024

2408.11593 2024-09-05 cs.MM cs.CV cs.SD eess.AS 82%

MCDubber: Multimodal Context-Aware Expressive Video Dubbing

Yuan Zhao, Zhenqi Jia, Rui Liu, De Hu, Feilong Bao, Guanglai Gao

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by NCMMSC2024

2408.14895 2024-08-29 cs.AI cs.CL cs.CV 82%

VHAKG: A Multi-modal Knowledge Graph Based on Synchronized Multi-view Videos of Daily Activities

Shusaku Egami, Takahiro Ugai, Swe Nwe Nwe Htun, Ken Fukuda

专题命中视频多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 5 pages, 4 figures, accepted by CIKM2024 Resource Track

2408.07694 2024-08-15 cs.CV cs.AI cs.LG cs.MM 82%

End-to-end Semantic-centric Video-based Multimodal Affective Computing

Ronghao Lin, Ying Zeng, Sijie Mai, Haifeng Hu

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Under Review

2401.10711 2024-07-24 cs.CV cs.AI cs.CL 82%

Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering

Haibo Wang, Chenghang Lai, Yixuan Sun, Weifeng Ge

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments accepted by ACM Multimedia 2024

2401.08743 2024-06-18 cs.AI cs.CL cs.CV cs.LG 82%

MMToM-QA: Multimodal Theory of Mind Question Answering

Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua B. Tenenbaum, Tianmin Shu

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL 2024. 26 pages, 11 figures, 7 tables

2406.06964 2024-06-12 cs.CL cs.MM cs.SD eess.AS 82%

Missingness-resilient Video-enhanced Multimodal Disfluency Detection

Payal Mohapatra, Shamika Likhite, Subrata Biswas, Bashima Islam, Qi Zhu

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted to Interspeech 2024

2405.16728 2024-05-28 cs.CV cs.AI cs.LG cs.MM 82%

Towards Multi-Task Multi-Modal Models: A Video Generative Perspective

Lijun Yu

专题命中视频多模态：multi-modal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments PhD thesis

2312.05291 2024-04-01 cs.CV cs.AI cs.CL 82%

GlitchBench: Can large multimodal models detect video game glitches?

Mohammad Reza Taesiri, Tianjun Feng, Anh Nguyen, Cor-Paul Bezemer

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2024

2312.02051 2024-03-29 cs.CV cs.AI cs.CL 82%

TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding

Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, Lu Hou

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2024 camera-ready version, code is available at https://github.com/RenShuhuai-Andy/TimeChat

2312.17172 2023-12-29 cs.CV cs.AI cs.CL 82%

Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action

Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, Aniruddha Kembhavi

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 38 pages, 20 figures

2312.00347 2023-12-19 cs.CV cs.CL cs.MM 82%

RTQ: Rethinking Video-language Understanding Based on Image-text Model

Xiao Wang, Yaoyu Li, Tian Gan, Zheng Zhang, Jingjing Lv, Liqiang Nie

专题命中视频多模态：image-text(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted by ACM MM 2023 as Oral representation

Journal ref In International Conference on Multimedia. ACM, 557--566 (2023)

2308.03741 2023-08-08 cs.CV cs.AI cs.LG cs.MM 82%

MAiVAR-T: Multimodal Audio-image and Video Action Recognizer using Transformers

Muhammad Bilal Shaikh, Douglas Chai, Syed Mohammed Shamsul Islam, Naveed Akhtar

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 6 pages, 7 figures, 4 tables, Peer reviewed, Accepted @ The 11th European Workshop on Visual Information Processing (EUVIP) will be held on 11th-14th September 2023, in Gjøvik, Norway. arXiv admin note: text overlap with arXiv:2103.15691 by other authors

2210.03625 2023-05-11 cs.CL cs.CV cs.MM 82%

C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval

Andrew Rouditchenko, Yung-Sung Chuang, Nina Shvetsova, Samuel Thomas, Rogerio Feris, Brian Kingsbury, Leonid Karlinsky, David Harwath, Hilde Kuehne, James Glass

专题命中视频多模态：cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted at ICASSP 2023. The code, models, and dataset are available at https://github.com/roudimit/c2kd

2305.03915 2023-05-09 cs.CV cs.CL cs.MM 82%

HateMM: A Multi-Modal Dataset for Hate Video Classification

Mithun Das, Rohit Raj, Punyajoy Saha, Binny Mathew, Manish Gupta, Animesh Mukherjee

专题命中视频多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted at ICWSM 2023(dataset track)

2208.08875 2023-04-12 physics.optics cs.GR 82%

Multi-modal imaging using a cascaded microscope design

Xi Yang, Mark Harfouche, Kevin C. Zhou, Lucas Kreiss, Shiqi Xu, Kanghyun Kim, Roarke Horstmeyer

专题命中视频多模态：multi-modal(title,abstract)；multimodal(abstract)

2207.01814 2023-01-23 cs.LG 82%

Multimodal Frame-Scoring Transformer for Video Summarization

Jeiyoon Park, Kiho Kwoun, Chanhee Lee, Heuiseok Lim

专题命中视频多模态：multimodal(title,abstract)；audio-visual(abstract)

Comments preprint

2210.08481 2022-10-18 cs.CV cs.CL cs.MM 82%

TLDW: Extreme Multimodal Summarisation of News Videos

Peggy Tang, Kun Hu, Lei Zhang, Jiebo Luo, Zhiyong Wang

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

2103.12541 2022-09-30 cs.MM cs.AI cs.CL cs.CR cs.CY cs.LG cs.SI 82%

A Survey on Multimodal Disinformation Detection

Firoj Alam, Stefano Cresci, Tanmoy Chakraborty, Fabrizio Silvestri, Dimiter Dimitrov, Giovanni Da San Martino, Shaden Shaar, Hamed Firooz, Preslav Nakov

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Accepted at COLING-2022, disinformation, misinformation, factuality, harmfulness, fake news, propaganda, multimodality, text, images, videos, network structure, temporality

2209.13537 2022-09-28 cs.NI cs.DB 82%

Sensing Multi-modal Mobility Patterns: A Case Study of Helsinki using Bluetooth Beacons and a Mobile Application

Zhiren Huang, Alonso Espinosa Mireles de Villafranca, Charalampos Sipetas

专题命中视频多模态：multi-modal(title,abstract)；multimodal(abstract)

2112.04446 2022-08-19 cs.CV cs.CL cs.SD eess.AS 82%

Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval

Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Rogerio Feris, David Harwath, James Glass, Hilde Kuehne

专题命中视频多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments CVPR2022. The final published version of the proceedings will be available on IEEE Xplore

2208.05814 2022-08-12 cs.CV cs.AI cs.MM 82%

Seeing your sleep stage: cross-modal distillation from EEG to infrared video

Jianan Han, Shaoxing Zhang, Aidong Men, Yang Liu, Ziming Yao, Yan Yan, Qingchao Chen

专题命中视频多模态：cross-modal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments We have submitted this paper to an academic journal

2206.07898 2022-06-17 cs.AI cs.CL cs.CV cs.LG 82%

Multimodal Dialogue State Tracking

Hung Le, Nancy F. Chen, Steven C. H. Hoi

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at NAACL 2022 (Oral)

2201.08264 2022-05-11 cs.CV cs.AI cs.CL cs.HC 82%

End-to-end Generative Pretraining for Multimodal Video Captioning

Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, Cordelia Schmid

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Journal ref Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) 2022

2204.03734 2022-04-11 cs.CV cs.CL cs.MM 82%

MHMS: Multimodal Hierarchical Multimedia Summarization

Jielin Qiu, Jiacheng Zhu, Mengdi Xu, Franck Dernoncourt, Trung Bui, Zhaowen Wang, Bo Li, Ding Zhao, Hailin Jin

专题命中视频多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments 10 pages

2112.03763 2022-02-03 cs.LG 82%

Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning

DeepMind Interactive Agents Team, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico Carnevale, Mary Cassin, Felix Fischer, Petko Georgiev, Alex Goldin, Mansi Gupta, Tim Harley, Felix Hill, Peter C Humphreys, Alden Hung, Jessica Landon, Timothy Lillicrap, Hamza Merzic, Alistair Muldal, Adam Santoro, Guy Scully, Tamara von Glehn, Greg Wayne, Nathaniel Wong, Chen Yan, Rui Zhu

专题命中视频多模态：multimodal(title,abstract)；multi-modal(abstract)

1. 视频多模态 4772 篇

Towards Expressive Video Dubbing with Multiscale Multimodal Context Interaction

Enhancing Multimodal Affective Analysis with Learned Live Comment Features

TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models

TV-TREES: Multimodal Entailment Trees for Neuro-Symbolic Video Reasoning

Enhancing Adverse Drug Event Detection with Multimodal Dataset: Corpus Creation and Model Development

MCDubber: Multimodal Context-Aware Expressive Video Dubbing

VHAKG: A Multi-modal Knowledge Graph Based on Synchronized Multi-view Videos of Daily Activities

End-to-end Semantic-centric Video-based Multimodal Affective Computing

Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering

MMToM-QA: Multimodal Theory of Mind Question Answering

Missingness-resilient Video-enhanced Multimodal Disfluency Detection

Towards Multi-Task Multi-Modal Models: A Video Generative Perspective

GlitchBench: Can large multimodal models detect video game glitches?

TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding

Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action

RTQ: Rethinking Video-language Understanding Based on Image-text Model

MAiVAR-T: Multimodal Audio-image and Video Action Recognizer using Transformers

C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval

HateMM: A Multi-Modal Dataset for Hate Video Classification

Multi-modal imaging using a cascaded microscope design

Multimodal Frame-Scoring Transformer for Video Summarization

TLDW: Extreme Multimodal Summarisation of News Videos

A Survey on Multimodal Disinformation Detection

Sensing Multi-modal Mobility Patterns: A Case Study of Helsinki using Bluetooth Beacons and a Mobile Application

Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval

Seeing your sleep stage: cross-modal distillation from EEG to infrared video

Multimodal Dialogue State Tracking

End-to-end Generative Pretraining for Multimodal Video Captioning

MHMS: Multimodal Hierarchical Multimedia Summarization

Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning