多模态大模型 - arXivDaily 专题

2407.05679 2025-05-01 cs.CV cs.AI 84%

BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents

Yumeng Zhang, Shi Gong, Kaixin Xiong, Xiaoqing Ye, Xiaofan Li, Xiao Tan, Fan Wang, Jizhou Huang, Hua Wu, Haifeng Wang

机构 * Baidu Inc.（百度公司）

专题命中多模态生成：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Comments 10 pages

2503.22941 2025-04-01 cs.AI cs.LG cs.MM 84%

Identifying Multi-modal Knowledge Neurons in Pretrained Transformers via Two-stage Filtering

Yugen Sato, Tomohiro Takagi

专题命中多模态生成：multi-modal(title)；multimodal(abstract)；MLLM(abstract)；分类 cs.AI、cs.MM

2503.10391 2025-03-14 cs.CV cs.AI 84%

CINEMA: Coherent Multi-Subject Video Generation via MLLM-Based Guidance

Yufan Deng, Xun Guo, Yizhi Wang, Jacob Zhiyuan Fang, Angtian Wang, Shenghai Yuan, Yiding Yang, Bo Liu, Haibin Huang, Chongyang Ma

专题命中多模态生成：MLLM(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

2502.11925 2025-03-11 cs.AI cs.CV cs.LG 84%

GRAPHGPT-O: Synergistic Multimodal Comprehension and Generation on Graphs

Yi Fang, Bowen Jin, Jiacheng Shen, Sirui Ding, Qiaoyu Tan, Jiawei Han

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

2406.09961 2025-03-03 cs.SE cs.CL cs.CV 84%

ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation

Cheng Yang, Chufan Shi, Yaxin Liu, Bo Shui, Junjie Wang, Mohan Jing, Linran Xu, Xinyu Zhu, Siheng Li, Yuxiang Zhang, Gongye Liu, Xiaomei Nie, Deng Cai, Yujiu Yang

专题命中多模态生成：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL

Comments Accepted to ICLR 2025. Data and code are available at https://github.com/ChartMimic/ChartMimic

2502.16469 2025-02-25 cs.CV cs.AI 84%

Cross-domain Few-shot Object Detection with Multi-modal Textual Enrichment

Zeyu Shangguan, Daniel Seita, Mohammad Rostami

专题命中多模态生成：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments arXiv admin note: substantial text overlap with arXiv:2403.16188

2502.14864 2025-02-21 cs.AI cs.CV 84%

Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Framework

Yuming Yang, Jiang Zhong, Li Jin, Jingwang Huang, Jingpeng Gao, Qing Liu, Yang Bai, Jingyuan Zhang, Rui Jiang, Kaiwen Wei

专题命中多模态生成：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

2405.10121 2025-02-06 cs.CL cs.MM 84%

Distilling Implicit Multimodal Knowledge into Large Language Models for Zero-Resource Dialogue Generation

Bo Zhang, Hui Ma, Jian Ding, Jian Wang, Bo Xu, Hongfei Lin

专题命中多模态生成：multimodal(title,abstract)；image-text(abstract)；分类 cs.CL、cs.MM

Comments Accepted by Information Fusion. The code is available at https://github.com/zhangbo-nlp/VIKDF

2502.00752 2025-02-04 cs.AI cs.CL cs.IR 84%

Zero-Shot Warning Generation for Misinformative Multimodal Content

Giovanni Pio Delvecchio, Huy Hong Nguyen, Isao Echizen

专题命中多模态生成：multimodal(title,abstract)；image-text(abstract)；分类 cs.CL、cs.AI

2412.05821 2024-12-11 cs.CL cs.AI 84%

An Entailment Tree Generation Approach for Multimodal Multi-Hop Question Answering with Mixture-of-Experts and Iterative Feedback Mechanism

Qing Zhang, Haocheng Lv, Jie Liu, Zhiyun Chen, Jianyong Duan, Hao Wang, Li He, Mingying Xv

专题命中多模态生成：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CL、cs.AI

Comments Erratum: We identified an error in the calculation of the F1 score in table 4 reported in a previous version of this work. The performance of the new result is better than the previous one. The corrected values are included in this updated version of the paper. These changes do not alter the primary conclusions of our research

Journal ref MM '2024: Proceedings of the 32nd ACM International Conference on Multimedia, Pages 4814 - 4822

2407.14242 2024-11-22 cs.CV cs.MM 84%

Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Images

Bo Yuan, Danpei Zhao, Zhuoran Liu, Wentao Li, Tian Li

专题命中多模态生成：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted in ACMMM 2024

2402.17304 2024-11-22 cs.CL cs.AI 84%

Probing Multimodal Large Language Models for Global and Local Semantic Representations

Mingxu Tao, Quzhe Huang, Kun Xu, Liwei Chen, Yansong Feng, Dongyan Zhao

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL、cs.AI

Comments Accepted by LREC-COLING 2024 as a short paper. ACL Anthology URL: [https://aclanthology.org/2024.lrec-main.1142/]

2411.11362 2024-11-19 cs.CV cs.CL 84%

MAIRA-Seg: Enhancing Radiology Report Generation with Segmentation-Aware Multimodal Large Language Models

Harshita Sharma, Valentina Salvatelli, Shaury Srivastav, Kenza Bouzid, Shruthi Bannur, Daniel C. Castro, Maximilian Ilse, Sam Bond-Taylor, Mercy Prasanna Ranjit, Fabian Falck, Fernando Pérez-García, Anton Schwaighofer, Hannah Richardson, Maria Teodora Wetscherek, Stephanie L. Hyland, Javier Alvarez-Valle

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL

Comments Accepted as Proceedings Paper at ML4H 2024

2410.17918 2024-10-24 cs.CV cs.AI cs.LG 84%

Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generation

Wenfang Yao, Chen Liu, Kejing Yin, William K. Cheung, Jing Qin

专题命中多模态生成：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by NeurIPS-24

2410.13754 2024-10-21 cs.AI cs.LG cs.MM 84%

MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures

Jinjie Ni, Yifan Song, Deepanway Ghosal, Bo Li, David Junhao Zhang, Xiang Yue, Fuzhao Xue, Zian Zheng, Kaichen Zhang, Mahir Shah, Kabir Jain, Yang You, Michael Shieh

专题命中多模态生成：any-to-any(title,abstract)；multi-modal(abstract)；分类 cs.AI、cs.MM

2406.16562 2024-10-11 cs.CV cs.CL 84%

EVALALIGN: Supervised Fine-Tuning Multimodal LLMs with Human-Aligned Data for Evaluating Text-to-Image Models

Zhiyu Tan, Xiaomeng Yang, Luozheng Qin, Mengping Yang, Cheng Zhang, Hao Li

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL

Comments Project page: https://sais-fuxi.github.io/projects/evalalign/

2404.09486 2024-09-27 cs.CL cs.CV cs.SE 84%

MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems

Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, Zhiyong Huang, Jing Ma

专题命中多模态生成：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

Comments EMNLP 2024

2403.00816 2024-08-15 cs.IR cs.AI cs.CV 84%

Read and Think: An Efficient Step-wise Multimodal Language Model for Document Understanding and Reasoning

Jinxu Zhang

专题命中多模态生成：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

2407.18854 2024-07-29 cs.CV cs.AI 84%

Unifying Visual and Semantic Feature Spaces with Diffusion Models for Enhanced Cross-Modal Alignment

Yuze Zheng, Zixuan Li, Xiangxian Li, Jinxing Liu, Yuqing Wang, Xiangxu Meng, Lei Meng

专题命中多模态生成：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

2401.11708 2024-06-05 cs.CV cs.AI cs.LG 84%

Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs

Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, Bin Cui

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments ICML 2024. Project: https://github.com/YangLing0818/RPG-DiffusionMaster

2207.07934 2024-05-14 cs.CL cs.HC cs.MM 84%

Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Model

Xiaolin Chen, Xuemeng Song, Liqiang Jing, Shuo Li, Linmei Hu, Liqiang Nie

专题命中多模态生成：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL、cs.MM

2310.02992 2024-04-29 cs.CV cs.CL 84%

Kosmos-G: Generating Images in Context with Multimodal Large Language Models

Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL

Comments Code: https://aka.ms/Kosmos-G Project Page: https://xichenpan.github.io/kosmosg

2404.08677 2024-04-16 cs.IR cs.AI cs.CL 84%

PMG : Personalized Multimodal Generation with Large Language Models

Xiaoteng Shen, Rui Zhang, Xiaoyan Zhao, Jieming Zhu, Xi Xiao

专题命中多模态生成：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CL、cs.AI

2404.02618 2024-04-04 cs.CV cs.AI 84%

Diffexplainer: Towards Cross-modal Global Explanations with Diffusion Models

Matteo Pennisi, Giovanni Bellitto, Simone Palazzo, Mubarak Shah, Concetto Spampinato

专题命中多模态生成：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

2308.12038 2024-03-25 cs.CL cs.CV 84%

Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages

Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, Maosong Sun

专题命中多模态生成：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments https://github.com/OpenBMB/VisCPM.git

2401.06805 2024-01-19 cs.CL cs.AI 84%

Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning

Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, Hongxia Yang

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL、cs.AI

2312.01367 2023-12-05 cs.CV cs.AI 84%

DiFace: Cross-Modal Face Recognition through Controlled Diffusion

Bowen Sun, Shibao Zheng

专题命中多模态生成：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

2311.16488 2023-11-29 cs.CV cs.AI 84%

Efficient Multimodal Diffusion Models Using Joint Data Infilling with Partially Shared U-Net

Zizhao Hu, Shaochong Jia, Mohammad Rostami

专题命中多模态生成：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

2304.14108 2023-10-23 cs.CV cs.CL cs.LG 84%

DataComp: In search of the next generation of multimodal datasets

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, Ludwig Schmidt

专题命中多模态生成：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments NeurIPS 2023 Datasets and Benchmarks Track

2310.05126 2023-10-10 cs.CV cs.AI 84%

UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model

Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Alex Lin, Fei Huang

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

1. 多模态生成 4959 篇

BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents

Identifying Multi-modal Knowledge Neurons in Pretrained Transformers via Two-stage Filtering

CINEMA: Coherent Multi-Subject Video Generation via MLLM-Based Guidance

GRAPHGPT-O: Synergistic Multimodal Comprehension and Generation on Graphs

ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation

Cross-domain Few-shot Object Detection with Multi-modal Textual Enrichment

Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Framework

Distilling Implicit Multimodal Knowledge into Large Language Models for Zero-Resource Dialogue Generation

Zero-Shot Warning Generation for Misinformative Multimodal Content

An Entailment Tree Generation Approach for Multimodal Multi-Hop Question Answering with Mixture-of-Experts and Iterative Feedback Mechanism

Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Images

Probing Multimodal Large Language Models for Global and Local Semantic Representations

MAIRA-Seg: Enhancing Radiology Report Generation with Segmentation-Aware Multimodal Large Language Models

Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generation

MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures

EVALALIGN: Supervised Fine-Tuning Multimodal LLMs with Human-Aligned Data for Evaluating Text-to-Image Models

MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems

Read and Think: An Efficient Step-wise Multimodal Language Model for Document Understanding and Reasoning

Unifying Visual and Semantic Feature Spaces with Diffusion Models for Enhanced Cross-Modal Alignment

Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs

Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Model

Kosmos-G: Generating Images in Context with Multimodal Large Language Models

PMG : Personalized Multimodal Generation with Large Language Models

Diffexplainer: Towards Cross-modal Global Explanations with Diffusion Models

Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages

Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning

DiFace: Cross-Modal Face Recognition through Controlled Diffusion

Efficient Multimodal Diffusion Models Using Joint Data Infilling with Partially Shared U-Net

DataComp: In search of the next generation of multimodal datasets

UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model