视觉大模型 / VLM

Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness

Khyathi Raghavi Chandu, Linjie Li, Anas Awadalla, Ximing Lu, Jae Sung Park, Jack Hessel, Lijuan Wang, Yejin Choi

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments 26 pages

Advancing Surgical VQA with Scene Graph Knowledge

Kun Yuan, Manasi Kattel, Joel L. Lavanchy, Nassir Navab, Vinkle Srivastav, Nicolas Padoy

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments IPCAI 2024, Int J CARS (2024)

MISS: A Generative Pretraining and Finetuning Approach for Med-VQA

Jiawei Chen, Dingkang Yang, Yue Jiang, Yuxuan Lei, Lihua Zhang

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments ICANN, 2024

KNVQA: A Benchmark for evaluation knowledge-based VQA

Sirui Cheng, Siyu Zhang, Jiayi Wu, Muchen Lan

专题命中 视觉问答 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments There was a little error in the method section of the paper

Lumos : Empowering Multimodal LLMs with Scene Text Recognition

Ashish Shenoy, Yichao Lu, Srihari Jayakumar, Debojeet Chatterjee, Mohsen Moslehpour, Pierce Chuang, Abhay Harpale, Vikas Bhardwaj, Di Xu, Shicong Zhao, Longfang Zhao, Ankit Ramchandani, Xin Luna Dong, Anuj Kumar

专题命中 视觉问答 ：multimodal large language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted to KDD 2024 (ADS Track)

Retrieval Augmented Generation for Domain-specific Question Answering

Sanat Sharma, David Seunghyun Yoon, Franck Dernoncourt, Dewang Sultania, Karishma Bagga, Mengjiao Zhang, Trung Bui, Varun Kotte

专题命中 视觉问答 ：grounding(abstract)；分类 cs.AI、cs.LG

Comments AAAI 2024 (Association for the Advancement of Artificial Intelligence) Scientific Document Understanding Workshop

EyeFound: A Multimodal Generalist Foundation Model for Ophthalmic Imaging

Danli Shi, Weiyi Zhang, Xiaolan Chen, Yexin Liu, Jiancheng Yang, Siyu Huang, Yih Chung Tham, Yingfeng Zheng, Mingguang He

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments 21 pages, 2 figures, 4 tables

Lost in Space: Probing Fine-grained Spatial Understanding in Vision and Language Resamplers

Georgios Pantazopoulos, Alessandro Suglia, Oliver Lemon, Arash Eshghi

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments NAACL 2024

Graph Neural Networks in Vision-Language Image Understanding: A Survey

Henry Senior, Gregory Slabaugh, Shanxin Yuan, Luca Rossi

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.LG

Comments 20 pages, 5 figures, 5 tables

Journal ref Vis Comput (2024)

FunQA: Towards Surprising Video Comprehension

Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hessel, Jingkang Yang, Ziwei Liu

专题命中 视觉问答 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Project Page: https://funqa-benchmark.github.io/ Codebase: https://github.com/Jingkang50/FunQA

Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Use

Imad Eddine Toubal, Aditya Avinash, Neil Gordon Alldrin, Jan Dlabal, Wenlei Zhou, Enming Luo, Otilia Stretcu, Hao Xiong, Chun-Ta Lu, Howard Zhou, Ranjay Krishna, Ariel Fuxman, Tom Duerig

专题命中 视觉问答 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Evaluating Very Long-Term Conversational Memory of LLM Agents

Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, Yuwei Fang

专题命中 视觉问答 ：grounding(abstract)；分类 cs.AI、cs.LG

Comments 19 pages; Project page: https://snap-research.github.io/locomo/

Multimodal Transformer With a Low-Computational-Cost Guarantee

Sungjin Park, Edward Choi

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.LG

Comments Accepted to ICASSP 2024 (5 pages)

MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editing

Jiaqi Li, Miaozeng Du, Chuanyi Zhang, Yongrui Chen, Nan Hu, Guilin Qi, Haiyun Jiang, Siyuan Cheng, Bozhong Tian

专题命中 视觉问答 ：multimodal large language model(abstract)；分类 cs.AI、cs.LG

Comments 8 pages

An Evaluation of GPT-4V and Gemini in Online VQA

Mengchen Liu, Chongyan Chen, Danna Gurari

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments 20 pages

Knowledge Generation for Zero-shot Knowledge-based VQA

Rui Cao, Jing Jiang

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments accepted as Findings in EACL 2023

On the Promises and Challenges of Multimodal Foundation Models for Geographical, Environmental, Agricultural, and Urban Planning Applications

Chenjiao Tan, Qian Cao, Yiwei Li, Jielu Zhang, Xiao Yang, Huaqin Zhao, Zihao Wu, Zhengliang Liu, Hao Yang, Nemin Wu, Tao Tang, Xinyue Ye, Lilong Chai, Ninghao Liu, Changying Li, Lan Mu, Tianming Liu, Gengchen Mai

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments 110 Pages; 61 Figures

EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images

Seongsu Bae, Daeun Kyung, Jaehee Ryu, Eunbyeol Cho, Gyubok Lee, Sunjun Kweon, Jungwoo Oh, Lei Ji, Eric I-Chao Chang, Tackeun Kim, Edward Choi

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Accepted at NeurIPS 2023 Datasets and Benchmarks Track (10 pages for main text, 4 pages for references, 39 pages for supplementary materials)

MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records

Scott L. Fleming, Alejandro Lozano, William J. Haberkorn, Jenelle A. Jindal, Eduardo P. Reis, Rahul Thapa, Louis Blankemeier, Julian Z. Genkins, Ethan Steinberg, Ashwin Nayak, Birju S. Patel, Chia-Chun Chiang, Alison Callahan, Zepeng Huo, Sergios Gatidis, Scott J. Adams, Oluseyi Fayanju, Shreya J. Shah, Thomas Savage, Ethan Goh, Akshay S. Chaudhari, Nima Aghaeepour, Christopher Sharp, Michael A. Pfeffer, Percy Liang, Jonathan H. Chen, Keith E. Morse, Emma P. Brunskill, Jason A. Fries, Nigam H. Shah

专题命中 视觉问答 ：grounding(abstract)；分类 cs.AI、cs.LG

Unleashing the Potential of Large Language Model: Zero-shot VQA for Flood Disaster Scenario

Yimin Sun, Chao Wang, Yan Peng

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments accepted by The 4th International Conference on Artificial Intelligence and Computer Engineering

The curse of language biases in remote sensing VQA: the role of spatial attributes, language diversity, and the need for clear evaluation

Christel Chappuis, Eliot Walt, Vincent Mendez, Sylvain Lobry, Bertrand Le Saux, Devis Tuia

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Zero-shot Translation of Attention Patterns in VQA Models to Natural Language

Leonard Salewski, A. Sophia Koepke, Hendrik P. A. Lensch, Zeynep Akata

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Published in GCPR 2023

Dynamic Task and Weight Prioritization Curriculum Learning for Multimodal Imagery

Huseyin Fuat Alsan, Taner Arsan

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

AVIS: Autonomous Visual Information Seeking with Large Language Model Agent

Ziniu Hu, Ahmet Iscen, Chen Sun, Kai-Wei Chang, Yizhou Sun, David A Ross, Cordelia Schmid, Alireza Fathi

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Published on NeurIPS 2023

Image Manipulation via Multi-Hop Instructions -- A New Dataset and Weakly-Supervised Neuro-Symbolic Approach

Harman Singh, Poorva Garg, Mohit Gupta, Kevin Shah, Ashish Goswami, Satyam Modi, Arnab Kumar Mondal, Dinesh Khandelwal, Dinesh Garg, Parag Singla

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments EMNLP 2023 (long paper, main conference)

tagE: Enabling an Embodied Agent to Understand Human Instructions

Chayan Sarkar, Avik Mitra, Pradip Pramanick, Tapas Nayak

专题命中 视觉问答 ：grounding(abstract)；分类 cs.AI、cs.LG

Comments Accepted in EMNLP Findings 2023

Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, Ming-Wei Chang

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments EMNLP 2023 (main conference); Our dataset and evaluation is available at https://open-vision-language.github.io/infoseek/

Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI

Leila Arras, Ahmed Osman, Wojciech Samek

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.LG

Comments 37 pages, 9 tables, 2 figures (plus appendix 14 pages)

Rad-ReStruct: A Novel VQA Benchmark and Method for Structured Radiology Reporting

Chantal Pellegrini, Matthias Keicher, Ege Özsoy, Nassir Navab

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments accepted at MICCAI 2023

Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP

Vedant Palit, Rohan Pandey, Aryaman Arora, Paul Pu Liang

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Final version for 5th Workshop on Closing the Loop Between Vision and Language (CLVL) @ ICCV 2023. 4 pages, 5 figures