多模态大模型 - arXivDaily 专题

2407.00569 2024-08-06 cs.CV cs.AI cs.CL 82%

Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models

Weihong Zhong, Xiaocheng Feng, Liang Zhao, Qiming Li, Lei Huang, Yuxuan Gu, Weitao Ma, Yuan Xu, Bing Qin

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to ACL 2024 Main Conference. 21 pages, 20 figures

2407.21174 2024-08-01 cs.CV cs.AI eess.AS 82%

AI Safety in Practice: Enhancing Adversarial Robustness in Multimodal Image Captioning

Maisha Binte Rashid, Pablo Rivas

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Accepted into KDD 2024 workshop on Ethical AI

2407.13559 2024-07-19 cs.CV cs.AI cs.CL 82%

Qalam : A Multimodal LLM for Arabic Optical Character and Handwriting Recognition

Gagan Bhatia, El Moatez Billah Nagoudi, Fakhraddin Alwajih, Muhammad Abdul-Mageed

专题命中图文多模态：multimodal(title)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

2407.00020 2024-07-02 cs.CV cs.AI cs.CL cs.IT cs.LG math.IT 82%

Visual Language Model based Cross-modal Semantic Communication Systems

Feibo Jiang, Chuanguo Tang, Li Dong, Kezhi Wang, Kun Yang, Cunhua Pan

专题命中图文多模态：cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 12 pages, 10 figures

2406.19736 2024-07-01 cs.CV cs.AI cs.CL cs.LG 82%

MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment

Jihao Liu, Xin Huang, Jinliang Zheng, Boxiao Liu, Jia Wang, Osamu Yoshie, Yu Liu, Hongsheng Li

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Dataset and models are available at https://github.com/jihaonew/MM-Instruct

2406.11048 2024-06-18 cs.LG cs.DC 82%

Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modality

Liwei Che, Jiaqi Wang, Xinyue Liu, Fenglong Ma

专题命中图文多模态：multi-modal(title,abstract)；image-text(abstract)

Comments Accepted by ECML-PKDD 2024

2406.06004 2024-06-11 cs.CV cs.AI cs.CL 82%

FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model

Yebin Lee, Imseong Park, Myungjoo Kang

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at ACL (Main) 2024

2405.01583 2024-05-06 cs.CL cs.AI cs.CV cs.LG 82%

MediFact at MEDIQA-M3G 2024: Medical Question Answering in Dermatology with Multimodal Learning

Nadia Saeed

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 7 pages, 3 figures, Clinical NLP 2024 workshop proceedings in Shared Task

2312.00784 2024-04-30 cs.CV cs.AI cs.CL cs.LG 82%

ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts

Mu Cai, Haotian Liu, Dennis Park, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Yong Jae Lee

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to CVPR2024. Project page: https://vip-llava.github.io/

2404.15653 2024-04-25 cs.CV cs.AI cs.CL cs.LG 82%

CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data

Sachin Mehta, Maxwell Horton, Fartash Faghri, Mohammad Hossein Sekhavat, Mahyar Najibi, Mehrdad Farajtabar, Oncel Tuzel, Mohammad Rastegari

专题命中图文多模态：image-text(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2404.15532 2024-04-25 cs.HC cs.AI cs.CL cs.CV cs.MA 82%

BattleAgent: Multi-modal Dynamic Emulation on Historical Battles to Complement Historical Analysis

Shuhang Lin, Wenyue Hua, Lingyao Li, Che-Jui Chang, Lizhou Fan, Jianchao Ji, Hang Hua, Mingyu Jin, Jiebo Luo, Yongfeng Zhang

专题命中图文多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 26 pages, 14 figures The data and code for this project are accessible at https://github.com/agiresearch/battleagent

2309.07915 2024-03-21 cs.CL cs.AI cs.CV 82%

MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning

Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, Baobao Chang

专题命中图文多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by ICLR2024

2403.06835 2024-03-12 cs.CV cs.AI cs.CL 82%

Medical Image Synthesis via Fine-Grained Image-Text Alignment and Anatomy-Pathology Prompting

Wenting Chen, Pengyu Wang, Hui Ren, Lichao Sun, Quanzheng Li, Yixuan Yuan, Xiang Li

专题命中图文多模态：image-text(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 5 figures

2402.18091 2024-02-29 cs.CV cs.AI cs.CL 82%

Polos: Multimodal Metric Learning from Human Feedback for Image Captioning

Yuiga Wada, Kanta Kaneda, Daichi Saito, Komei Sugiura

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2024

2310.03211 2023-11-29 cs.CL cs.AI cs.CV 82%

On the Performance of Multimodal Language Models

Utsav Garg, Erhan Bas

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2310.18369 2023-10-31 cs.CL cs.AI cs.CV 82%

Apollo: Zero-shot MultiModal Reasoning with Multiple Experts

Daniela Ben-David, Tzuf Paz-Argaman, Reut Tsarfaty

专题命中图文多模态：multimodal(title)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments GitHub: https://github.com/danielabd/Apollo-Cap

2302.05608 2023-10-09 cs.CV cs.AI cs.CL cs.LG 82%

Differentiable Outlier Detection Enable Robust Deep Multimodal Analysis

Zhu Wang, Sourav Medya, Sathya N. Ravi

专题命中图文多模态：multimodal(title)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

2308.15226 2023-08-30 cs.CV cs.AI cs.CL 82%

CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation

Devaansh Gupta, Siddhant Kharbanda, Jiawei Zhou, Wanhua Li, Hanspeter Pfister, Donglai Wei

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 15 pages, 9 figures, to be published In Proceedings of International Conference of Computer Vision(ICCV), 2023

2209.06997 2022-09-16 cs.LG cs.CR 82%

M^4I: Multi-modal Models Membership Inference

Pingyi Hu, Zihan Wang, Ruoxi Sun, Hu Wang, Minhui Xue

专题命中图文多模态：multi-modal(title,abstract)；multimodal(abstract)

Comments Accepted to NeurIPS 2022

2204.00598 2022-05-30 cs.CV cs.AI cs.CL cs.LG 82%

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, Pete Florence

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments https://socraticmodels.github.io/

2204.07955 2022-04-22 cs.CV cs.CL cs.MM 82%

Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis

Yan Ling, Jianfei Yu, Rui Xia

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted by ACL 2022 (long paper)

2201.11517 2022-01-28 q-bio.NC cs.LG cs.NE 82%

Multimodal neural networks better explain multivoxel patterns in the hippocampus

Bhavin Choksi, Milad Mozafari, Rufin VanRullen, Leila Reddy

专题命中图文多模态：multimodal(title,abstract)；multi-modal(abstract)

Comments Oral at SVRHM Workshop (NeurIPS 2021)

2010.08725 2020-10-20 cs.CL cs.AI cs.CV 82%

A Corpus for English-Japanese Multimodal Neural Machine Translation with Comparable Sentences

Andrew Merritt, Chenhui Chu, Yuki Arase

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2009.11278 2020-09-24 cs.CV cs.AI cs.CL cs.LG 82%

X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers

Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannaneh Hajishirzi, Aniruddha Kembhavi

专题命中图文多模态：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments EMNLP 2020

1905.12260 2020-07-02 cs.CL cs.AI cs.CV cs.LG 82%

Learning Multilingual Word Embeddings Using Image-Text Data

Karan Singhal, Karthik Raman, Balder ten Cate

专题命中图文多模态：image-text(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2004.14973 2020-05-04 cs.CV cs.AI cs.CL cs.LG 82%

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web

Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, Dhruv Batra

专题命中图文多模态：image-text(title,abstract)；分类 cs.CV、cs.CL、cs.AI

1907.04378 2019-07-11 cs.CV cs.CL cs.LG eess.AS eess.IV 82%

M3D-GAN: Multi-Modal Multi-Domain Translation with Universal Attention

Shuang Ma, Daniel McDuff, Yale Song

专题命中图文多模态：multi-modal(title)；cross-modal(abstract)；分类 cs.CV、cs.CL、eess.AS

1906.06892 2019-06-18 cs.CV cs.CL cs.LG cs.MM 82%

ParNet: Position-aware Aggregated Relation Network for Image-Text matching

Yaxian Xia, Lun Huang, Wenmin Wang, Xiaoyong Wei, Wenmin Wang

专题命中图文多模态：image-text(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments 9 pages, 3 figures

1811.10092 2019-04-09 cs.CV cs.AI cs.CL cs.RO 82%

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, Lei Zhang

专题命中图文多模态：cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2019 Oral

1712.03404 2017-12-12 cs.IR 82%

Semi-supervised Multimodal Hashing

Dayong Tian, Maoguo Gong, Deyun Zhou, Jiao Shi, Yu Lei

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)

1. 图文多模态 4657 篇

Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models

AI Safety in Practice: Enhancing Adversarial Robustness in Multimodal Image Captioning

Qalam : A Multimodal LLM for Arabic Optical Character and Handwriting Recognition

Visual Language Model based Cross-modal Semantic Communication Systems

MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment

Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modality

FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model

MediFact at MEDIQA-M3G 2024: Medical Question Answering in Dermatology with Multimodal Learning

ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts

CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data

BattleAgent: Multi-modal Dynamic Emulation on Historical Battles to Complement Historical Analysis

MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning

Medical Image Synthesis via Fine-Grained Image-Text Alignment and Anatomy-Pathology Prompting

Polos: Multimodal Metric Learning from Human Feedback for Image Captioning

On the Performance of Multimodal Language Models

Apollo: Zero-shot MultiModal Reasoning with Multiple Experts

Differentiable Outlier Detection Enable Robust Deep Multimodal Analysis

CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation

M^4I: Multi-modal Models Membership Inference

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis

Multimodal neural networks better explain multivoxel patterns in the hippocampus

A Corpus for English-Japanese Multimodal Neural Machine Translation with Comparable Sentences

X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers

Learning Multilingual Word Embeddings Using Image-Text Data

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web

M3D-GAN: Multi-Modal Multi-Domain Translation with Universal Attention

ParNet: Position-aware Aggregated Relation Network for Image-Text matching

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Semi-supervised Multimodal Hashing