视觉大模型 / VLM

How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model

Shezheng Song, Xiaopeng Li, Shasha Li, Shan Zhao, Jie Yu, Jun Ma, Xiaoguang Mao, Weimin Zhang

机构 * College of Computer, National University of Defense Technology（国防科技大学计算机学院） ； School of Computer and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院） 纠错

专题命中 视觉问答 ：multimodal large language model(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted by TKDE

VQA$^2$: Visual Question Answering for Video Quality Assessment

Ziheng Jia, Zicheng Zhang, Jiaying Qian, Haoning Wu, Wei Sun, Chunyi Li, Xiaohong Liu, Weisi Lin, Guangtao Zhai, Xiongkuo Min

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.AI

Comments 23 pages 12 figures

Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?

Daniel P. Jeong, Saurabh Garg, Zachary C. Lipton, Michael Oberst

专题命中 视觉问答 ：vision-language model(title,abstract)；分类 cs.AI、cs.LG

Comments This version was published at EMNLP 2024 Main Conference as a Long Paper (Oral). See the extended version (arXiv:2411.08870) for additional results on QA tasks based on clinical notes and evaluations in the supervised fine-tuning regime

ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering

Yifan Wu, Lutao Yan, Leixian Shen, Yunhai Wang, Nan Tang, Yuyu Luo

专题命中 视觉问答 ：multimodal large language model(title,abstract)；分类 cs.CV、cs.AI

Comments EMNLP 2024 Conference Paper

D3: Data Diversity Design for Systematic Generalization in Visual Question Answering

Amir Rahimi, Vanessa D'Amario, Moyuru Yamada, Kentaro Takemoto, Tomotake Sasaki, Xavier Boix

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.AI

Comments TMLR (https://openreview.net/forum?id=ZAin13msOp)

Dreaming Out Loud: A Self-Synthesis Approach For Training Vision-Language Models With Developmentally Plausible Data

Badr AlKhamissi, Yingtian Tang, Abdülkadir Gökce, Johannes Mehrer, Martin Schrimpf

专题命中 视觉问答 ：vision-language model(title)；visual question answering(abstract)；分类 cs.CV、cs.LG

Comments Accepted to BabyLM Challenge at CoNLL 2024

From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities

Md Farhan Ishmam, Md Sakib Hossain Shovon, M. F. Mridha, Nilanjan Dey

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.AI

Visual Question Answering in Ophthalmology: A Progressive and Practical Perspective

Xiaolan Chen, Ruoyu Chen, Pusheng Xu, Weiyi Zhang, Xianwen Shang, Mingguang He, Danli Shi

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.AI

RSAdapter: Adapting Multimodal Models for Remote Sensing Visual Question Answering

Yuduo Wang, Pedram Ghamisi

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.LG

Journal ref IEEE Trans. Geosci. Remote Sens., vol. 62, pp. 1-13, 2024

Dragonfly: Multi-Resolution Zoom-In Encoding Enhances Vision-Language Models

Rahul Thapa, Kezhen Chen, Ian Covert, Rahul Chalamala, Ben Athiwaratkun, Shuaiwen Leon Song, James Zou

专题命中 视觉问答 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Eliminating the Language Bias for Visual Question Answering with fine-grained Causal Intervention

Ying Liu, Ge Bai, Chenji Lu, Shilong Li, Zhang Zhang, Ruifang Liu, Wenbin Guo

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.AI

Journal ref 2024 IEEE International Conference on Multimedia and Expo (ICME), Niagara Falls, ON, Canada, 2024, pp. 1-6

Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contexts

Aditya Sharma, Michael Saxon, William Yang Wang

专题命中 视觉问答 ：vision language model(title,abstract)；分类 cs.CV、cs.AI

Comments Findings of EMNLP 2024

YesBut: A High-Quality Annotated Multimodal Dataset for evaluating Satire Comprehension capability of Vision-Language Models

Abhilash Nandy, Yash Agarwal, Ashish Patwa, Millon Madhur Das, Aman Bansal, Ankit Raj, Pawan Goyal, Niloy Ganguly

专题命中 视觉问答 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Comments EMNLP 2024 Main (Long), 18 pages, 14 figures, 12 tables

MAPWise: Evaluating Vision-Language Models for Advanced Map Queries

Srija Mukhopadhyay, Abhishek Rajgaria, Prerana Khatiwada, Vivek Gupta, Dan Roth

专题命中 视觉问答 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Comments 30 Pages, 46 Tables, 6 Figure

Knowledge Detection by Relevant Question and Image Attributes in Visual Question Answering

Param Ahir, Hiteishi Diwanji

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.AI

IIU: Independent Inference Units for Knowledge-based Visual Question Answering

Yili Li, Jing Yu, Keke Gai, Gang Xiong

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.AI

Target Prompting for Information Extraction with Vision Language Model

Dipankar Medhi

专题命中 视觉问答 ：vision language model(title,abstract)；分类 cs.CV、cs.AI

Comments 7 pages, 5 figures

Boosting Audio Visual Question Answering via Key Semantic-Aware Cues

Guangyao Li, Henghui Du, Di Hu

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted by ACM MM 2024

AdaCoder: Adaptive Prompt Compression for Programmatic Visual Question Answering

Mahiro Ukai, Shuhei Kurita, Atsushi Hashimoto, Yoshitaka Ushiku, Nakamasa Inoue

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.AI

Learning Trimodal Relation for Audio-Visual Question Answering with Missing Modality

Kyu Ri Park, Hong Joo Lee, Jung Uk Kim

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted at ECCV 2024

Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering

Haibo Wang, Chenghang Lai, Yixuan Sun, Weifeng Ge

专题命中 视觉问答 ：grounding(title,abstract)；分类 cs.CV、cs.AI

Comments accepted by ACM Multimedia 2024

TM-PATHVQA:90000+ Textless Multilingual Questions for Medical Visual Question Answering

Tonmoy Rajkhowa, Amartya Roy Chowdhury, Sankalp Nagaonkar, Achyut Mani Tripathi

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.AI

Comments conference (Accepted at Interspeech 2024)

FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding

Huitong Pan, Qi Zhang, Cornelia Caragea, Eduard Dragut, Longin Jan Latecki

专题命中 视觉问答 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Comments ECAI 2024

WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answering

Pingyi Chen, Chenglu Zhu, Sunyi Zheng, Honglin Li, Lin Yang

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted at ECCV 2024

Towards Multilingual Audio-Visual Question Answering

Orchid Chetia Phukan, Priyabrata Mallick, Swarup Ranjan Behera, Aalekhya Satya Narayani, Arun Balaji Buduru, Rajesh Sharma

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.LG

Comments Accepted to Interspeech 2024

Revisiting the Role of Language Priors in Vision-Language Models

Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, Deva Ramanan

专题命中 视觉问答 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Comments Published at ICML 2024. Website: https://linzhiqiu.github.io/papers/visual_gpt_score/

TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domains

Yoonsik Kim, Moonbin Yim, Ka Yeon Song

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.AI

Comments Technical Report

Enhancing Visual Question Answering through Question-Driven Image Captions as Prompts

Övgü Özdemir, Erdem Akagündüz

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.AI

Comments The paper has been accepted for presentation at CVPR 2024 Workshop on Prompting in Vision

Instance-Level Trojan Attacks on Visual Question Answering via Adversarial Learning in Neuron Activation Space

Yuwei Sun, Hideya Ochiai, Jun Sakuma

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted for IJCNN 2024

Adversarial Training with OCR Modality Perturbation for Scene-Text Visual Question Answering

Zhixuan Shen, Haonan Luo, Sijia Li, Tianrui Li

专题命中 视觉问答 ：visual question answering(title,abstract)；分类 cs.CV、cs.AI

Comments 6 pages, 3 figures, accepted by 2024 IEEE International Conference on Multimedia and Expo