视觉大模型 / VLM

GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI

Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, Shaoting Zhang, Bin Fu, Jianfei Cai, Bohan Zhuang, Eric J Seibel, Junjun He, Yu Qiao

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV

Comments GitHub: https://github.com/uni-medical/GMAI-MMBench Hugging face: https://huggingface.co/datasets/OpenGVLab/GMAI-MMBench

OMCAT: Omni Context Aware Transformer

Arushi Goel, Karan Sapra, Matthieu Le, Rafael Valle, Andrew Tao, Bryan Catanzaro

专题命中 视觉问答 ：visual question answering(abstract)；grounding(abstract)；分类 cs.CV

Comments Demo page: https://om-cat.github.io

Difficult Task Yes but Simple Task No: Unveiling the Laziness in Multimodal LLMs

Sihang Zhao, Youliang Yuan, Xiaoying Tang, Pinjia He

专题命中 视觉问答 ：LLaVA(abstract)；multimodal large language model(abstract)；分类 cs.AI

Comments EMNLP 2024 Findings

NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training

Yiyi Tao, Zhuoyue Wang, Hang Zhang, Lun Wang

专题命中 视觉问答 ：vision language model(abstract)；visual question answering(abstract)；分类 cs.CV

MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting

Fangchen Liu, Kuan Fang, Pieter Abbeel, Sergey Levine

专题命中 视觉问答 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

PropTest: Automatic Property Testing for Improved Visual Programming

Jaywon Koo, Ziyan Yang, Paola Cascante-Bonilla, Baishakhi Ray, Vicente Ordonez

专题命中 视觉问答 ：visual reasoning(abstract)；visual question answering(abstract)；分类 cs.CV

Comments Project Page: https://jaywonkoo17.github.io/PropTest/

How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts

Yusu Qian, Haotian Zhang, Yinfei Yang, Zhe Gan

专题命中 视觉问答 ：LLaVA(abstract)；multimodal large language model(abstract)；分类 cs.CV

GPT-4V Explorations: Mining Autonomous Driving

Zixuan Li

专题命中 视觉问答 ：visual language model(abstract)；visual question answering(abstract)；分类 cs.CV

Instruction Makes a Difference

Tosin Adewumi, Nudrat Habib, Lama Alkhaled, Elisa Barney

专题命中 视觉问答 ：LLaVA(abstract)；visual question answering(abstract)；分类 cs.CV

Comments Accepted at the 16th IAPR International Workshop On Document Analysis Systems (DAS)

Understanding Information Storage and Transfer in Multi-modal Large Language Models

Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, Soheil Feizi, Daniela Massiceti

专题命中 视觉问答 ：LLaVA(abstract)；visual question answering(abstract)；分类 cs.CV

Comments 20 pages

Hawk: Learning to Understand Open-World Video Anomalies

Jiaqi Tang, Hao Lu, Ruizheng Wu, Xiaogang Xu, Ke Ma, Cheng Fang, Bin Guo, Jiangbo Lu, Qifeng Chen, Ying-Cong Chen

专题命中 视觉问答 ：VLM(abstract)；visual language model(abstract)；分类 cs.CV

OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM

Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, Ping Luo

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV

Language-Informed Visual Concept Learning

Sharon Lee, Yunzhi Zhang, Shangzhe Wu, Jiajun Wu

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV

Comments ICLR 2024. The first two authors contributed equally and are alphabetically ordered. Project page: https://ai.stanford.edu/~yzzhang/projects/concept-axes/

WoLF: Wide-scope Large Language Model Framework for CXR Understanding

Seil Kang, Donghyun Kim, Junhyeok Kim, Hyo Kyung Lee, Seong Jae Hwang

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.AI

Comments 11 pages main paper, 2 pages supplementary

From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models

Dongsheng Jiang, Yuchen Liu, Songlin Liu, Jin'e Zhao, Hao Zhang, Zhen Gao, Xiaopeng Zhang, Jin Li, Hongkai Xiong

专题命中 视觉问答 ：visual question answering(abstract)；grounding(abstract)；分类 cs.CV

Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications

Muhammad Arslan Manzoor, Sarah Albarri, Ziting Xian, Zaiqiao Meng, Preslav Nakov, Shangsong Liang

专题命中 视觉问答 ：visual reasoning(abstract)；visual question answering(abstract)；分类 cs.AI

SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark

Zhenwen Liang, Kehan Guo, Gang Liu, Taicheng Guo, Yujun Zhou, Tianyu Yang, Jiajun Jiao, Renjie Pi, Jipeng Zhang, Xiangliang Zhang

专题命中 视觉问答 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.AI

Comments Work in progress

LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding

Senqiao Yang, Jiaming Liu, Ray Zhang, Mingjie Pan, Zoey Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Yandong Guo, Shanghang Zhang

专题命中 视觉问答 ：grounding(abstract)；multimodal large language model(abstract)；分类 cs.CV

NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotations

Yuichi Inoue, Yuki Yada, Kotaro Tanahashi, Yu Yamaguchi

专题命中 视觉问答 ：vision language model(abstract)；visual question answering(abstract)；分类 cs.CV

Comments Accepted at LLVM-AD Workshop @ WACV 2024

Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamics Audio-Visual Scenarios

Yuanyuan Jiang, Jianqin Yin

专题命中 视觉问答 ：visual question answering(abstract)；grounding(abstract)；分类 cs.CV

Comments Accepted to EMNLP 2023 Findings

LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models

Yanwei Li, Chengyao Wang, Jiaya Jia

专题命中 视觉问答 ：vision language model(abstract)；visual question answering(abstract)；分类 cs.CV

Comments Code is available at https://github.com/dvlab-research/LLaMA-VID

MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning

Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, Mohamed Elhoseiny

专题命中 视觉问答 ：visual question answering(abstract)；grounding(abstract)；分类 cs.CV

Learning to Follow Object-Centric Image Editing Instructions Faithfully

Tuhin Chakrabarty, Kanishk Singh, Arkadiy Saakyan, Smaranda Muresan

专题命中 视觉问答 ：visual question answering(abstract)；grounding(abstract)；分类 cs.CV

Comments Findings of EMNLP 2023 (Long paper)

Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4V

Zhiling Yan, Kai Zhang, Rong Zhou, Lifang He, Xiang Li, Lichao Sun

专题命中 视觉问答 ：visual question answering(abstract)；multimodal large language model(abstract)；分类 cs.CV

JourneyDB: A Benchmark for Generative Image Understanding

Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, Limin Wang, Hongsheng Li

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV

Comments Accepted to the Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023)

Exploring Question Decomposition for Zero-Shot VQA

Zaid Khan, Vijay Kumar BG, Samuel Schulter, Manmohan Chandraker, Yun Fu

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV

Comments NeurIPS 2023 Camera Ready

Uni-NLX: Unifying Textual Explanations for Vision and Vision-Language Tasks

Fawaz Sammani, Nikos Deligiannis

专题命中 视觉问答 ：visual reasoning(abstract)；visual question answering(abstract)；分类 cs.CV

Comments Accepted to ICCVW 2023

Imaginations of WALL-E : Reconstructing Experiences with an Imagination-Inspired Module for Advanced AI Systems

Zeinab Sadat Taghavi, Soroush Gooran, Seyed Arshan Dalili, Hamidreza Amirzadeh, Mohammad Jalal Nematbakhsh, Hossein Sameti

专题命中 视觉问答 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.AI

Comments 18 pages,

Let's ViCE! Mimicking Human Cognitive Behavior in Image Generation Evaluation

Federico Betti, Jacopo Staiano, Lorenzo Baraldi, Lorenzo Baraldi, Rita Cucchiara, Nicu Sebe

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV

Comments Accepted as oral at ACM MultiMedia 2023 (Brave New Ideas track)

mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections

Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, Luo Si

专题命中 视觉问答 ：visual question answering(abstract)；grounding(abstract)；分类 cs.CV

Journal ref EMNLP2022