多模态大模型

4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities

Roman Bachmann, Oğuzhan Fatih Kar, David Mizrahi, Ali Garjani, Mingfei Gao, David Griffiths, Jiaming Hu, Afshin Dehghan, Amir Zamir

专题命中 多模态生成 ：any-to-any(title)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments Project page at 4m.epfl.ch

TRINS: Towards Multimodal Language Models that Can Read

Ruiyi Zhang, Yanzhe Zhang, Jian Chen, Yufan Zhou, Jiuxiang Gu, Changyou Chen, Tong Sun

专题命中 多模态生成 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments CVPR 2024

PostDoc: Generating Poster from a Long Multimodal Document Using Deep Submodular Optimization

Vijay Jaisankar, Sambaran Bandyopadhyay, Kalp Vyas, Varre Chaitanya, Shwetha Somasundaram

专题命中 多模态生成 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspective

Zihao Yue, Liang Zhang, Qin Jin

专题命中 多模态生成 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted to ACL 2024

CamViG: Camera Aware Image-to-Video Generation with Multimodal Transformers

Andrew Marmon, Grant Schindler, José Lezama, Dan Kondratyuk, Bryan Seybold, Irfan Essa

专题命中 多模态生成 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Hijacking Context in Large Multi-modal Models

Joonhyun Jeong

专题命中 多模态生成 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Comments Technical Report. Preprint

Journal ref ICLR 2024 Workshop on Reliable and Responsible Foundation Models

FashionSD-X: Multimodal Fashion Garment Synthesis using Latent Diffusion

Abhishek Kumar Singh, Ioannis Patras

专题命中 多模态生成 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments 9 pages, 8 figures

Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation

Xun Wu, Shaohan Huang, Furu Wei

专题命中 多模态生成 ：multimodal(title,abstract)；分类 cs.CV、cs.MM

Universal Fingerprint Generation: Controllable Diffusion Model with Multimodal Conditions

Steven A. Grosz, Anil K. Jain

专题命中 多模态生成 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Model

Chunyi Li, Guo Lu, Donghui Feng, Haoning Wu, Zicheng Zhang, Xiaohong Liu, Guangtao Zhai, Weisi Lin, Wenjun Zhang

专题命中 多模态生成 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments 13 page, 11 figures, 4 tables

PAIR-Diffusion: A Comprehensive Multimodal Object-Level Image Editor

Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Xingqian Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang, Humphrey Shi

专题命中 多模态生成 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted in CVPR 2024, Project page https://vidit98.github.io/publication/conference-paper/pair_diff.html

Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision

Seongyun Lee, Sue Hyun Park, Yongrae Jo, Minjoon Seo

专题命中 多模态生成 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Memory-based Cross-modal Semantic Alignment Network for Radiology Report Generation

Yitian Tao, Liyan Ma, Jing Yu, Han Zhang

专题命中 多模态生成 ：cross-modal(title,abstract)；分类 cs.CV、cs.AI

Comments 12 pages, 8 figures

MM-Diff: High-Fidelity Image Personalization via Multi-Modal Condition Integration

Zhichao Wei, Qingkun Su, Long Qin, Weizhi Wang

专题命中 多模态生成 ：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.AI

Prompt Highlighter: Interactive Control for Multi-Modal LLMs

Yuechen Zhang, Shengju Qian, Bohao Peng, Shu Liu, Jiaya Jia

专题命中 多模态生成 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Comments CVPR 2024; Project Page: https://julianjuaner.github.io/projects/PromptHighlighter

TopicDiff: A Topic-enriched Diffusion Approach for Multimodal Conversational Emotion Detection

Jiamin Luo, Jingjing Wang, Guodong Zhou

专题命中 多模态生成 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

A Multimodal In-Context Tuning Approach for E-Commerce Product Description Generation

Yunxin Li, Baotian Hu, Wenhan Luo, Lin Ma, Yuxin Ding, Min Zhang

专题命中 多模态生成 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted by LREC-COLING 2024

RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis

Yao Mu, Junting Chen, Qinglong Zhang, Shoufa Chen, Qiaojun Yu, Chongjian Ge, Runjian Chen, Zhixuan Liang, Mengkang Hu, Chaofan Tao, Peize Sun, Haibao Yu, Chao Yang, Wenqi Shao, Wenhai Wang, Jifeng Dai, Yu Qiao, Mingyu Ding, Ping Luo

专题命中 多模态生成 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations

Qizhi Pei, Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingce Xia, Rui Yan

专题命中 多模态生成 ：cross-modal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by Empirical Methods in Natural Language Processing 2023 (EMNLP 2023)

Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models

Hongzhan Lin, Ziyang Luo, Wei Gao, Jing Ma, Bo Wang, Ruichao Yang

专题命中 多模态生成 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments The first work towards explainable harmful meme detection by harnessing advanced LLMs

Journal ref The ACM Web Conference 2024

Text-to-Image Cross-Modal Generation: A Systematic Review

Maciej Żelaszczyk, Jacek Mańdziuk

专题命中 多模态生成 ：cross-modal(title,abstract)；分类 cs.CV、cs.CL

Yes, this is what I was looking for! Towards Multi-modal Medical Consultation Concern Summary Generation

Abhisek Tiwari, Shreyangshu Bera, Sriparna Saha, Pushpak Bhattacharyya, Samrat Ghosh

专题命中 多模态生成 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clients

DaiXun Li, Weiying Xie, ZiXuan Wang, YiBing Lu, Yunsong Li, Leyuan Fang

专题命中 多模态生成 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation

Marco Bellagente, Manuel Brack, Hannah Teufel, Felix Friedrich, Björn Deiseroth, Constantin Eichenberg, Andrew Dai, Robert Baldock, Souradeep Nanda, Koen Oostermeijer, Andres Felipe Cruz-Salinas, Patrick Schramowski, Kristian Kersting, Samuel Weinbach

专题命中 多模态生成 ：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments Proceedings of Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS)

VLIS: Unimodal Language Models Guide Multimodal Language Generation

Jiwan Chung, Youngjae Yu

专题命中 多模态生成 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted as main paper in EMNLP 2023

4M: Massively Multimodal Masked Modeling

David Mizrahi, Roman Bachmann, Oğuzhan Fatih Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, Amir Zamir

专题命中 多模态生成 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments NeurIPS 2023 Spotlight. Project page at https://4m.epfl.ch/

Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis

Chaoyi Wu, Jiayu Lei, Qiaoyu Zheng, Weike Zhao, Weixiong Lin, Xiaoman Zhang, Xiao Zhou, Ziheng Zhao, Ya Zhang, Yanfeng Wang, Weidi Xie

专题命中 多模态生成 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

ChartLlama: A Multimodal LLM for Chart Understanding and Generation

Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, Hanwang Zhang

专题命中 多模态生成 ：multimodal(title)；multi-modal(abstract)；分类 cs.CV、cs.CL

Comments Code and model on https://tingxueronghua.github.io/ChartLlama/

GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation

Can Qin, Ning Yu, Chen Xing, Shu Zhang, Zeyuan Chen, Stefano Ermon, Yun Fu, Caiming Xiong, Ran Xu

专题命中 多模态生成 ：multi-modal(title,abstract)；分类 cs.CV、cs.MM

Comments ICCV 2023

Multimodal Representations for Teacher-Guided Compositional Visual Reasoning

Wafa Aissa, Marin Ferecatu, Michel Crucianu

专题命中 多模态生成 ：multimodal(title)；cross-modal(abstract)；分类 cs.CV、cs.CL

Journal ref Advanced Concepts for Intelligent Vision Systems, 21st International Conference (ACIVS 2023), Aug 2023, Kumamoto, Japan