多模态大模型

What matters when building vision-language models?

Hugo Laurençon, Léo Tronchon, Matthieu Cord, Victor Sanh

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Learning Hierarchical Image Segmentation For Recognition and By Recognition

Tsung-Wei Ke, Sangwoo Mo, Stella X. Yu

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.AI

Comments ICLR 2024 (spotlight). First two authors contributed equally. Code available at https://github.com/twke18/CAST

Exploring the Distinctiveness and Fidelity of the Descriptions Generated by Large Vision-Language Models

Yuhang Huang, Zihan Wu, Chongyang Gao, Jiawei Peng, Xu Yang

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.MM

Comments 11 pages, 9 figures, 6 tables. For associated code, see https://anonymous.4open.science/r/Explore_FGVDs-E277

Language Models as Knowledge Bases for Visual Word Sense Disambiguation

Anastasia Kritharoula, Maria Lymperaiou, Giorgos Stamou

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI

Journal ref KBC-LM workshop@ ISWC 2023

VG4D: Vision-Language Model Goes 4D Video Recognition

Zhichao Deng, Xiangtai Li, Xia Li, Yunhai Tong, Shen Zhao, Mengyuan Liu

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.AI

Comments ICRA 2024

Evolving Interpretable Visual Classifiers with Large Language Models

Mia Chiquier, Utkarsh Mall, Carl Vondrick

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

ChimpVLM: Ethogram-Enhanced Chimpanzee Behaviour Recognition

Otto Brookes, Majid Mirmehdi, Hjalmar Kuhl, Tilo Burghardt

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI

SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples

Phillip Howard, Avinash Madasu, Tiep Le, Gustavo Lujan Moreno, Anahita Bhiwandiwalla, Vasudev Lal

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.AI

Comments Accepted to CVPR 2024. arXiv admin note: text overlap with arXiv:2310.02988

OmniFusion Technical Report

Elizaveta Goncharova, Anton Razzhigaev, Matvey Mikhalchuk, Maxim Kurkin, Irina Abdullaeva, Matvey Skripkin, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments 17 pages, 4 figures, 9 tables, 2 appendices

Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning

Zijie Song, Zhenzhen Hu, Yuanen Zhou, Ye Zhao, Richang Hong, Meng Wang

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.MM

Unleash the Potential of CLIP for Video Highlight Detection

Donghoon Han, Seunghyeon Seo, Eunhwan Park, Seong-Uk Nam, Nojun Kwak

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Residual-based Language Models are Free Boosters for Biomedical Imaging

Zhixin Lai, Jing Wu, Suiyao Chen, Yucheng Zhou, Naira Hovakimyan

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Toward Interactive Regional Understanding in Vision-Large Language Models

Jungbeom Lee, Sanghyuk Chun, Sangdoo Yun

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL

Comments NAACL 2024 Main Conference

TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification

Qinying Liu, Wei Wu, Kecheng Zheng, Zhan Tong, Jiawei Liu, Yu Liu, Wei Chen, Zilei Wang, Yujun Shen

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.AI

HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data

Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, Yueting Zhuang

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by CVPR 2024

TiC-CLIP: Continual Training of CLIP Models

Saurabh Garg, Mehrdad Farajtabar, Hadi Pouransari, Raviteja Vemulapalli, Sachin Mehta, Oncel Tuzel, Vaishaal Shankar, Fartash Faghri

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL

Comments ICLR 2024

Few-Shot VQA with Frozen LLMs: A Tale of Two Approaches

Igor Sterner, Weizhe Lin, Jinghong Chen, Bill Byrne

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

VeCLIP: Improving CLIP Training via Visual-enriched Captions

Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, Meng Cao

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.AI

Comments CV/ML

Actional Atomic-Concept Learning for Demystifying Vision-Language Navigation

Bingqian Lin, Yi Zhu, Xiaodan Liang, Liang Lin, Jianzhuang Liu

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by AAAI 2023

Language-Driven Anchors for Zero-Shot Adversarial Robustness

Xiao Li, Wei Zhang, Yining Liu, Zhanhao Hu, Bo Zhang, Xiaolin Hu

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by CVPR 2024

Alt-Text with Context: Improving Accessibility for Images on Twitter

Nikita Srivatsan, Sofia Samaniego, Omar Florez, Taylor Berg-Kirkpatrick

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments ICLR 2024

On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study

Minh-Hao Van, Prateek Verma, Xintao Wu

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing

Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Pérez-García, Maximilian Ilse, Daniel C. Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, Anton Schwaighofer, Maria Wetscherek, Matthew P. Lungren, Aditya Nori, Javier Alvarez-Valle, Ozan Oktay

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments To appear in CVPR 2023

Vision-Language Interpreter for Robot Task Planning

Keisuke Shirai, Cristian C. Beltran-Hernandez, Masashi Hamaya, Atsushi Hashimoto, Shohei Tanaka, Kento Kawaharazuka, Kazutoshi Tanaka, Yoshitaka Ushiku, Shinsuke Mori

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments ICRA 2024

Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning

Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, Lifu Huang

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments 8 Pages, visual instruction tuning

Visual In-Context Learning for Large Vision-Language Models

Yucheng Zhou, Xiang Li, Qianning Wang, Jianbing Shen

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL

Comments 13 pages, 7 figures

VIGC: Visual Instruction Generation and Correction

Bin Wang, Fan Wu, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiaoyi Dong, Weijia Li, Wei Li, Jiaqi Wang, Conghui He

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by AAAI 2024, Project Website: https://opendatalab.github.io/VIGC, Code and Pretrained Model: https://github.com/opendatalab/VIGC

LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding

Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, Tong Sun

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments Preprint

Parrot Captions Teach CLIP to Spot Text

Yiqi Lin, Conghui He, Alex Jinpeng Wang, Bin Wang, Weijia Li, Mike Zheng Shou

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.AI

Comments project page: https://linyq17.github.io/CLIP-Parrot-Bias/. Add more analysis and ablation studies. Update Figure 3 with a more precise metric

ClipSAM: CLIP and SAM Collaboration for Zero-Shot Anomaly Segmentation

Shengze Li, Jianjian Cao, Peng Ye, Yuhan Ding, Chongjun Tu, Tao Chen

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.AI

Comments 17 pages,17 figures