视觉大模型 / VLM

Leveraging Large Language Models for Scalable Vector Graphics-Driven Image Understanding

Mu Cai, Zeyi Huang, Yuheng Li, Utkarsh Ojha, Haohan Wang, Yong Jae Lee

专题命中 视觉问答 ：visual reasoning(abstract)；分类 cs.CV、cs.AI、cs.LG

Composition Vision-Language Understanding via Segment and Depth Anything Model

Mingxiao Huo, Pengliang Ji, Haotian Lin, Junchen Liu, Yixiao Wang, Yijun Chen

专题命中 视觉问答 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Reverse Image Retrieval Cues Parametric Memory in Multimodal LLMs

Jialiang Xu, Michael Moor, Jure Leskovec

专题命中 视觉问答 ：visual question answering(abstract)；multimodal large language model(abstract)

SCULPT: Shape-Conditioned Unpaired Learning of Pose-dependent Clothed and Textured Human Meshes

Soubhik Sanyal, Partha Ghosh, Jinlong Yang, Michael J. Black, Justus Thies, Timo Bolkart

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Updated to camera ready version of CVPR 2024

Advancing Multimodal Medical Capabilities of Gemini

Lin Yang, Shawn Xu, Andrew Sellergren, Timo Kohlberger, Yuchen Zhou, Ira Ktena, Atilla Kiraly, Faruk Ahmed, Farhad Hormozdiari, Tiam Jaroensri, Eric Wang, Ellery Wulczyn, Fayaz Jamil, Theo Guidroz, Chuck Lau, Siyuan Qiao, Yun Liu, Akshay Goel, Kendall Park, Arnav Agharwal, Nick George, Yang Wang, Ryutaro Tanno, David G. T. Barrett, Wei-Hung Weng, S. Sara Mahdavi, Khaled Saab, Tao Tu, Sreenivasa Raju Kalidindi, Mozziyar Etemadi, Jorge Cuadros, Gregory Sorensen, Yossi Matias, Katherine Chou, Greg Corrado, Joelle Barral, Shravya Shetty, David Fleet, S. M. Ali Eslami, Daniel Tse, Shruthi Prabhakara, Cory McLean, Dave Steiner, Rory Pilgrim, Christopher Kelly, Shekoofeh Azizi, Daniel Golden

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation

Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, Su Wang

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments ICLR 2024; Project website: https://google.github.io/dsg

Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey

Zhuo Chen, Yichi Zhang, Yin Fang, Yuxia Geng, Lingbing Guo, Xiang Chen, Qian Li, Wen Zhang, Jiaoyan Chen, Yushan Zhu, Jiaqi Li, Xiaoze Liu, Jeff Z. Pan, Ningyu Zhang, Huajun Chen

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Ongoing work; 41 pages (Main Text), 55 pages (Total), 11 Tables, 13 Figures, 619 citations; Paper list is available at https://github.com/zjukg/KG-MM-Survey

Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology

Nur Yildirim, Hannah Richardson, Maria T. Wetscherek, Junaid Bajwa, Joseph Jacob, Mark A. Pinnock, Stephen Harris, Daniel Coelho de Castro, Shruthi Bannur, Stephanie L. Hyland, Pratik Ghosh, Mercy Ranjit, Kenza Bouzid, Anton Schwaighofer, Fernando Pérez-García, Harshita Sharma, Ozan Oktay, Matthew Lungren, Javier Alvarez-Valle, Aditya Nori, Anja Thieme

专题命中 视觉问答 ：vision-language model(abstract)；VLM(abstract)

Comments to appear at CHI 2024

Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models

Didi Zhu, Zhongyi Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Kun Kuang, Chao Wu

专题命中 视觉问答 ：LLaVA(abstract)；visual question answering(abstract)

Improving Automatic VQA Evaluation Using Large Language Models

Oscar Mañas, Benno Krojer, Aishwarya Agrawal

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted at AAAI 2024 (main track)

VQA-based Robotic State Recognition Optimized with Genetic Algorithm

Kento Kawaharazuka, Yoshiki Obinata, Naoaki Kanazawa, Kei Okada, Masayuki Inaba

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)

Comments Accepted at ICRA2023

Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning

Enna Sachdeva, Nakul Agarwal, Suhas Chundi, Sean Roelofs, Jiachen Li, Mykel Kochenderfer, Chiho Choi, Behzad Dariush

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

UnICLAM:Contrastive Representation Learning with Adversarial Masking for Unified and Interpretable Medical Vision Question Answering

Chenlu Zhan, Peng Peng, Hongsen Wang, Tao Chen, Hongwei Wang

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought

Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, Ping Luo

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

DLIP: Distilling Language-Image Pre-training

Huafeng Kuang, Jie Wu, Xiawu Zheng, Ming Li, Xuefeng Xiao, Rui Wang, Min Zheng, Rongrong Ji

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

I2I: Initializing Adapters with Improvised Knowledge

Tejas Srinivasan, Furong Jia, Mohammad Rostami, Jesse Thomason

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted at 2nd Conference on Lifelong Learning Agents (CoLLAs), 2023

GRILL: Grounded Vision-language Pre-training via Aligning Text and Image Regions

Woojeong Jin, Subhabrata Mukherjee, Yu Cheng, Yelong Shen, Weizhu Chen, Ahmed Hassan Awadallah, Damien Jose, Xiang Ren

专题命中 视觉问答 ：visual question answering(abstract)；grounding(abstract)

Comments Preprint

A Study of Autoregressive Decoders for Multi-Tasking in Computer Vision

Lucas Beyer, Bo Wan, Gagan Madan, Filip Pavetic, Andreas Steiner, Alexander Kolesnikov, André Susano Pinto, Emanuele Bugliarello, Xiao Wang, Qihang Yu, Liang-Chieh Chen, Xiaohua Zhai

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Optimizing Explanations by Network Canonization and Hyperparameter Search

Frederik Pahde, Galip Ümit Yolcu, Alexander Binder, Wojciech Samek, Sebastian Lapuschkin

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Few-shot Multimodal Multitask Multilingual Learning

Aman Chadha, Vinija Jain

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting

Oscar Mañas, Pau Rodriguez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, Aishwarya Agrawal

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted at EACL 2023 (main track); 26 pages, 21 figures, 6 tables; Pau Rodriguez and Saba Ahmadi had equal contributions

SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models

Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, Animesh Garg

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted by ICLR 2023. Project page: https://slotformer.github.io/

CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks

Zhecan Wang, Noel Codella, Yen-Chun Chen, Luowei Zhou, Jianwei Yang, Xiyang Dai, Bin Xiao, Haoxuan You, Shih-Fu Chang, Lu Yuan

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments This paper is greatly modified and updated to be re-submitted to another conference. The new paper is under the name "Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Tasks", https://doi.org/10.48550/arXiv.2204.10496

AI2-THOR: An Interactive 3D Environment for Visual AI

Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, Aniruddha Kembhavi, Abhinav Gupta, Ali Farhadi

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers

Estelle Aflalo, Meng Du, Shao-Yen Tseng, Yongfei Liu, Chenfei Wu, Nan Duan, Vasudev Lal

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Best Demo Award at CVPR 2022

American == White in Multimodal Language-and-Image AI

Robert Wolfe, Aylin Caliskan

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted to AI Ethics and Society 2022

VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Tasks

Hung Le, Nancy F. Chen, Steven C. H. Hoi

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted at NAACL 2022 (Oral)

Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Tasks

Zhecan Wang, Noel Codella, Yen-Chun Chen, Luowei Zhou, Xiyang Dai, Bin Xiao, Jianwei Yang, Haoxuan You, Kai-Wei Chang, Shih-fu Chang, Lu Yuan

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments arXiv admin note: substantial text overlap with arXiv:2201.05729

Towards General Purpose Vision Systems

Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, Derek Hoiem

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments CVPR 2022 Oral; Project page: https://prior.allenai.org/projects/gpv

Co-VQA : Answering by Interactive Sub Question Sequence

Ruonan Wang, Yuxi Qian, Fangxiang Feng, Xiaojie Wang, Huixing Jiang

专题命中 视觉问答 ：visual reasoning(abstract)；visual question answering(abstract)

Comments Accepted by Findings of ACL 2022