多模态大模型 - arXivDaily 专题

2407.11315 2024-07-17 cs.AI 83%

COMET: "Cone of experience" enhanced large multimodal model for mathematical problem generation

Sannyuya Liu, Jintian Feng, Zongkai Yang, Yawei Luo, Qian Wan, Xiaoxuan Shen, Jianwen Sun

专题命中多模态生成：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI

2407.07518 2024-07-11 cs.CV 83%

Multi-modal Crowd Counting via a Broker Modality

Haoliang Meng, Xiaopeng Hong, Chenhao Wang, Miao Shang, Wangmeng Zuo

专题命中多模态生成：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments This is the preprint version of the paper and supplemental material to appear in ECCV 2024. Please cite the final published version. Code is available at https://github.com/HenryCilence/Broker-Modality-Crowd-Counting

2407.05980 2024-07-09 cs.CV 83%

MMIS: Multimodal Dataset for Interior Scene Visual Generation and Recognition

Hozaifa Kassab, Ahmed Mahmoud, Mohamed Bahaa, Ammar Mohamed, Ali Hamdi

专题命中多模态生成：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

2406.07686 2024-06-13 cs.CV 83%

AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation

Kai Wang, Shijian Deng, Jing Shi, Dimitrios Hatzinakos, Yapeng Tian

专题命中多模态生成：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

2311.07470 2024-06-12 cs.CL 83%

Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers

Haowen Pan, Yixin Cao, Xiaozhi Wang, Xun Yang, Meng Wang

专题命中多模态生成：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CL

2402.12451 2024-06-07 cs.CV cs.AI cs.CL cs.MM 83%

The Revolution of Multimodal Large Language Models: A Survey

Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, Rita Cucchiara

专题命中多模态生成：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL 2024 (Findings)

2402.14859 2024-06-04 cs.CR cs.AI cs.CY cs.LG 83%

The Wolf Within: Covert Injection of Malice into MLLM Societies via an MLLM Operative

Zhen Tan, Chengshuai Zhao, Raha Moraffah, Yifan Li, Yu Kong, Tianlong Chen, Huan Liu

专题命中多模态生成：MLLM(title,abstract)；multimodal(abstract)；分类 cs.AI

Comments Accepted to workshop on ReGenAI@CVPR 2024

2405.01926 2024-05-06 cs.CV 83%

Auto-Encoding Morph-Tokens for Multimodal LLM

Kaihang Pan, Siliang Tang, Juncheng Li, Zhaoyu Fan, Wei Chow, Shuicheng Yan, Tat-Seng Chua, Yueting Zhuang, Hanwang Zhang

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments Accepted by ICML 2024

2404.05674 2024-04-09 cs.CV 83%

MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation

Kunpeng Song, Yizhe Zhu, Bingchen Liu, Qing Yan, Ahmed Elgammal, Xiao Yang

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

2312.06988 2023-12-19 cs.CV 83%

MWSIS: Multimodal Weakly Supervised Instance Segmentation with 2D Box Annotations for Autonomous Driving

Guangfeng Jiang, Jun Liu, Yuzhi Wu, Wenlong Liao, Tao He, Pai Peng

专题命中多模态生成：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments AAAI2024

2312.06739 2023-12-13 cs.CV 83%

SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models

Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, Ying Shan

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments Project page: https://yuzhou914.github.io/SmartEdit/

2312.06685 2023-12-13 cs.AI 83%

Causal-CoG: A Causal-Effect Look at Context Generation for Boosting Multi-modal Language Models

Shitian Zhao, Zhuowan Li, Yadong Lu, Alan Yuille, Yan Wang

专题命中多模态生成：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.AI

2312.01771 2023-12-05 cs.CV 83%

IMProv: Inpainting-based Multimodal Prompting for Computer Vision Tasks

Jiarui Xu, Yossi Gandelsman, Amir Bar, Jianwei Yang, Jianfeng Gao, Trevor Darrell, Xiaolong Wang

专题命中多模态生成：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Project page: https://jerryxu.net/IMProv

2312.01745 2023-12-05 cs.CV 83%

Cross-Modal Adaptive Dual Association for Text-to-Image Person Retrieval

Dixuan Lin, Yixing Peng, Jingke Meng, Wei-Shi Zheng

专题命中多模态生成：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

2311.17618 2023-12-04 cs.CV 83%

ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model

Fukun Yin, Xin Chen, Chi Zhang, Biao Jiang, Zibo Zhao, Jiayuan Fan, Gang Yu, Taihao Li, Tao Chen

专题命中多模态生成：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

2311.18835 2023-12-01 cs.CV 83%

InstructSeq: Unifying Vision Tasks with Instruction-conditioned Multi-modal Sequence Generation

Rongyao Fang, Shilin Yan, Zhaoyang Huang, Jingqiu Zhou, Hao Tian, Jifeng Dai, Hongsheng Li

专题命中多模态生成：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments 10 pages

2311.00618 2023-11-02 cs.CV 83%

De-Diffusion Makes Text a Strong Cross-Modal Interface

Chen Wei, Chenxi Liu, Siyuan Qiao, Zhishuai Zhang, Alan Yuille, Jiahui Yu

专题命中多模态生成：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Technical report. Project page: https://dediffusion.github.io

2310.07222 2023-10-12 cs.CV 83%

Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model

Shiyuan Yang, Xiaodong Chen, Jing Liao

专题命中多模态生成：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by ACMMM'23

2308.09306 2023-08-21 cs.CV 83%

DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability

Runhui Huang, Jianhua Han, Guansong Lu, Xiaodan Liang, Yihan Zeng, Wei Zhang, Hang Xu

专题命中多模态生成：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments ICCV2023

2306.17115 2023-07-04 cs.CV 83%

Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation

Zibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng, Rui Wang, Pei Cheng, Bin Fu, Tao Chen, Gang Yu, Shenghua Gao

专题命中多模态生成：image-text(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Project Website: https://neuralcarver.github.io/michelangelo

2306.06583 2023-06-13 cs.CV 83%

REACT2023: the first Multi-modal Multiple Appropriate Facial Reaction Generation Challenge

Siyang Song, Micol Spitale, Cheng Luo, German Barquero, Cristina Palmero, Sergio Escalera, Michel Valstar, Tobias Baur, Fabien Ringeval, Elisabeth Andre, Hatice Gunes

专题命中多模态生成：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV

2305.18433 2023-05-31 cs.LG cs.CV 83%

Cognitively Inspired Cross-Modal Data Generation Using Diffusion Models

Zizhao Hu, Mohammad Rostami

专题命中多模态生成：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

2303.06555 2023-05-31 cs.LG cs.CV 83%

One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale

Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, Jun Zhu

专题命中多模态生成：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted to ICML2023

2212.09478 2023-03-27 cs.CV 83%

MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation

Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, Baining Guo

专题命中多模态生成：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by CVPR 2023

2209.03430 2023-02-21 cs.LG cs.AI cs.CL cs.CV cs.MM 83%

Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency

专题命中多模态生成：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2206.07771 2023-02-17 cs.CV 83%

Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation

Ye Zhu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, Yan Yan

专题命中多模态生成：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments ICLR 2023. Project at https://github.com/L-YeZhu/CDCD

2212.09918 2022-12-21 cs.AI 83%

Generalizing Multimodal Variational Methods to Sets

Jinzhao Zhou, Yiqun Duan, Zhihong Chen, Yu-Cheng Chang, Chin-Teng Lin

专题命中多模态生成：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI

Comments First Submission

2206.13393 2022-07-15 eess.IV cs.CV cs.LG q-bio.NC 83%

Cross-Modal Transformer GAN: A Brain Structure-Function Deep Fusing Framework for Alzheimer's Disease

Junren Pan, Shuqiang Wang

专题命中多模态生成：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

2207.03430 2022-07-08 eess.IV cs.CV 83%

A Novel Unified Conditional Score-based Generative Framework for Multi-modal Medical Image Completion

Xiangxi Meng, Yuning Gu, Yongsheng Pan, Nizhuan Wang, Peng Xue, Mengkang Lu, Xuming He, Yiqiang Zhan, Dinggang Shen

专题命中多模态生成：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2101.04702 2022-04-15 cs.CV 83%

Cross-Modal Contrastive Learning for Text-to-Image Generation

Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, Yinfei Yang

专题命中多模态生成：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments CVPR 2021

1. 多模态生成 4959 篇

COMET: "Cone of experience" enhanced large multimodal model for mathematical problem generation

Multi-modal Crowd Counting via a Broker Modality

MMIS: Multimodal Dataset for Interior Scene Visual Generation and Recognition

AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation

Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers

The Revolution of Multimodal Large Language Models: A Survey

The Wolf Within: Covert Injection of Malice into MLLM Societies via an MLLM Operative

Auto-Encoding Morph-Tokens for Multimodal LLM

MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation

MWSIS: Multimodal Weakly Supervised Instance Segmentation with 2D Box Annotations for Autonomous Driving

SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models

Causal-CoG: A Causal-Effect Look at Context Generation for Boosting Multi-modal Language Models

IMProv: Inpainting-based Multimodal Prompting for Computer Vision Tasks

Cross-Modal Adaptive Dual Association for Text-to-Image Person Retrieval

ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model

InstructSeq: Unifying Vision Tasks with Instruction-conditioned Multi-modal Sequence Generation

De-Diffusion Makes Text a Strong Cross-Modal Interface

Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model

DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability

Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation

REACT2023: the first Multi-modal Multiple Appropriate Facial Reaction Generation Challenge

Cognitively Inspired Cross-Modal Data Generation Using Diffusion Models

One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale

MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation

Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation

Generalizing Multimodal Variational Methods to Sets

Cross-Modal Transformer GAN: A Brain Structure-Function Deep Fusing Framework for Alzheimer's Disease

A Novel Unified Conditional Score-based Generative Framework for Multi-modal Medical Image Completion

Cross-Modal Contrastive Learning for Text-to-Image Generation