视觉大模型 / VLM

How Well Can Vison-Language Models Understand Humans' Intention? An Open-ended Theory of Mind Question Evaluation Benchmark

Ximing Wen, Mallika Mainali, Anik Sen

机构 * College of Computing and Informatics, Drexel University（计算与信息学院，德雷塞尔大学） 纠错

专题命中 视觉问答 ：vision language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments 4 pages, accepted by ToM@AAAI25

VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents

Ryota Tanaka, Taichi Iki, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito, Jun Suzuki

机构 * NTT Human Informatics Laboratories, NTT Corporation（NTT公司人类信息学研究所） ； Tohoku University（东北大学） 纠错

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Accepted by CVPR 2025; project page: https://vdocrag.github.io

NoTeS-Bank: Benchmarking Neural Transcription and Search for Scientific Notes Understanding

Aniket Pal, Sanket Biswas, Alloy Das, Ayush Lodh, Priyanka Banerjee, Soumitri Chattopadhyay, Dimosthenis Karatzas, Josep Llados, C. V. Jawahar

机构 * IIIT Hyderabad（印度信息技术研究所海得拉巴分校） ； Computer Vision Center, UAB（巴塞罗那自治大学计算机视觉中心） ； Habitat Lens Pvt. Ltd.（栖息地镜头私人有限公司） ； UNC Chapel Hill（北卡罗来纳大学教堂山分校） 纠错

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.LG

MetaToken: Detecting Hallucination in Image Descriptions by Meta Classification

Laura Fieback, Jakob Spiegelberg, Hanno Gottschalk

机构 * Volkswagen AG（大众汽车集团） ； TU Berlin（柏林工业大学） 纠错

专题命中 视觉问答 ：vision language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.LG

GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosis

Bo Liu, Ke Zou, Liming Zhan, Zexin Lu, Xiaoyu Dong, Yidi Chen, Chengqiang Xie, Jiannong Cao, Xiao-Ming Wu, Huazhu Fu

机构 * The Hong Kong Polytechnic University（香港理工大学） ； National University of Singapore（新加坡国立大学） ； Sichuan University（四川大学） ； West China Hospital of Sichuan University（四川大学华西医院） ； Institute of High Performance Computing, Agency for Science, Technology and Research（新加坡科技研究局高性能计算研究所） 纠错

专题命中 视觉问答 ：vision language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments This project is available at https://www.med-vqa.com/GEMeX

Robusto-1 Dataset: Comparing Humans and VLMs on real out-of-distribution Autonomous Driving VQA from Peru

Dunant Cusipuma, David Ortega, Victor Flores-Benites, Arturo Deza

机构 * Artificio 纠错

专题命中 视觉问答 ：visual language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments A pre-print. 26 pages. Link to Code + Data: https://huggingface.co/datasets/Artificio/robusto-1

PreMind: Multi-Agent Video Understanding for Advanced Indexing of Presentation-style Videos

Kangda Wei, Zhengyu Zhou, Bingqing Wang, Jun Araki, Lukas Lange, Ruihong Huang, Zhe Feng

机构 * Bosch Research North America（博世北美研究院） ； Bosch Center for Artificial Intelligence (BCAI)（博世人工智能中心） ； Department of Computer Science and Engineering, Texas A&M University（德克萨斯农工大学计算机科学与工程学院） 纠错

专题命中 视觉问答 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Understanding Figurative Meaning through Explainable Visual Entailment

Arkadiy Saakyan, Shreyas Kulkarni, Tuhin Chakrabarty, Smaranda Muresan

机构 * Columbia University（哥伦比亚大学） 纠错

专题命中 视觉问答 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments NAACL 2025 Main Conference

MoColl: Agent-Based Specific and General Model Collaboration for Image Captioning

Pu Yang, Bin Dong

机构 * New Cornerstone Science Laboratory（新基石科学实验室） 纠错

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Consistency of Compositional Generalization across Multiple Levels

Chuanhao Li, Zhen Li, Chenchen Jing, Xiaomeng Fan, Wenbo Ye, Yuwei Wu, Yunde Jia

专题命中 视觉问答 ：visual question answering(abstract)；grounding(abstract)；分类 cs.CV、cs.AI

Comments Accepted by AAAI 2025

CEGI: Measuring the trade-off between efficiency and carbon emissions for SLMs and VLMs

Abhas Kumar, Kapil Pathak, Rajesh Kavuru, Prabhakar Srinivasan

专题命中 视觉问答 ：vision language model(abstract)；visual question answering(abstract)；分类 cs.AI、cs.LG

Enhancing Instruction-Following Capability of Visual-Language Models by Reducing Image Redundancy

Te Yang, Jian Jia, Xiangyu Zhu, Weisong Zhao, Bo Wang, Yanhua Cheng, Yan Li, Shengyuan Liu, Quan Chen, Peng Jiang, Kun Gai, Zhen Lei

专题命中 视觉问答 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answering

Guohao Sun, Can Qin, Huazhu Fu, Linwei Wang, Zhiqiang Tao

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.LG

Comments EMNLP 2024 Main conference

Mind the Uncertainty in Human Disagreement: Evaluating Discrepancies between Model Predictions and Human Responses in VQA

Jian Lan, Diego Frassinelli, Barbara Plank

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Parameter Efficient Reinforcement Learning from Human Feedback

Hakim Sidahmed, Samrat Phatale, Alex Hutcheson, Zhuonan Lin, Zhang Chen, Zac Yu, Jarvis Jin, Simral Chaudhary, Roman Komarytsia, Christiane Ahlheim, Yonghao Zhu, Bowen Li, Saravanan Ganesh, Bill Byrne, Jessica Hoffmann, Hassan Mansoor, Wei Li, Abhinav Rastogi, Lucas Dixon

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.AI、cs.LG

Multi-Modal Instruction-Tuning Small-Scale Language-and-Vision Assistant for Semiconductor Electron Micrograph Analysis

Sakhinana Sagar Srinivas, Geethan Sannidhi, Venkataramana Runkana

专题命中 视觉问答 ：visual question answering(abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.LG

Comments Paper published at AAAI 2024 Spring Symposium Series

Extracting Training Data from Document-Based VQA Models

Francesco Pinto, Nathalie Rauschmayr, Florian Tramèr, Philip Torr, Federico Tombari

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.LG

Comments ICML 2024

WeatherQA: Can Multimodal Language Models Reason about Severe Weather?

Chengqian Ma, Zhanxiang Hua, Alexandra Anderson-Frey, Vikram Iyer, Xin Liu, Lianhui Qin

专题命中 视觉问答 ：vision language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments 26 pages, 9 figures

AlanaVLM: A Multimodal Embodied AI Foundation Model for Egocentric Video Understanding

Alessandro Suglia, Claudio Greco, Katie Baker, Jose L. Part, Ioannis Papaioannou, Arash Eshghi, Ioannis Konstas, Oliver Lemon

专题命中 视觉问答 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments Code available https://github.com/alanaai/EVUD

Wings: Learning Multimodal LLMs without Text-only Forgetting

Yi-Kai Zhang, Shiyin Lu, Yang Li, Yanqing Ma, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, Han-Jia Ye

专题命中 视觉问答 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.AI、cs.LG

Bridging Vision and Language Spaces with Assignment Prediction

Jungin Park, Jiyoung Lee, Kwanghoon Sohn

专题命中 视觉问答 ：visual question answering(abstract)；grounding(abstract)；分类 cs.CV、cs.LG

Comments ICLR 2024 Camera-ready

Can I Trust Your Answer? Visually Grounded Video Question Answering

Junbin Xiao, Angela Yao, Yicong Li, Tat Seng Chua

专题命中 视觉问答 ：vision-language model(abstract)；grounding(abstract)；分类 cs.CV、cs.AI

Comments Accepted to CVPR'24. (Compared with preprint version, we mainly improve the presentation, discuss more related works, and extend experiments in Appendix.)

A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis

Yingshu Li, Yunyi Liu, Zhanyu Wang, Xinyu Liang, Lei Wang, Lingqiao Liu, Leyang Cui, Zhaopeng Tu, Longyue Wang, Luping Zhou

专题命中 视觉问答 ：visual question answering(abstract)；grounding(abstract)；分类 cs.CV、cs.AI

MIVC: Multiple Instance Visual Component for Visual-Language Models

Wenyi Wu, Qi Li, Wenliang Zhong, Junzhou Huang

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Accepted at WACV 2024

DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedback

Jiao Sun, Deqing Fu, Yushi Hu, Su Wang, Royi Rassin, Da-Cheng Juan, Dana Alon, Charles Herrmann, Sjoerd van Steenkiste, Ranjay Krishna, Cyrus Rashtchian

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs

Shengzhi Li, Nima Tajbakhsh

专题命中 视觉问答 ：LLaVA(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Med-Flamingo: a Multimodal Medical Few-shot Learner

Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Cyril Zakka, Yash Dalmia, Eduardo Pontes Reis, Pranav Rajpurkar, Jure Leskovec

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Preprint

Is Multimodal Vision Supervision Beneficial to Language?

Avinash Madasu, Vasudev Lal

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory

Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A. Ross, Alireza Fathi

专题命中 视觉问答 ：visual language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Published on CVPR 2023

Towards Models that Can See and Read

Roy Ganz, Oren Nuriel, Aviad Aberdam, Yair Kittenplon, Shai Mazor, Ron Litman

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.LG