多模态大模型 - arXivDaily 专题

2406.19578 2024-07-01 cs.CV cs.AI cs.CL cs.LG 67%

PathAlign: A vision-language model for whole slide images in histopathology

Faruk Ahmed, Andrew Sellergren, Lin Yang, Shawn Xu, Boris Babenko, Abbi Ward, Niels Olson, Arash Mohtashamian, Yossi Matias, Greg S. Corrado, Quang Duong, Dale R. Webster, Shravya Shetty, Daniel Golden, Yun Liu, David F. Steiner, Ellery Wulczyn

专题命中图文多模态：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 9 main pages and 19 pages of supplemental material; 3 main tables, 3 main figures and 11 supplemental tables, 7 supplemental figures

2405.20204 2024-06-27 cs.CL cs.AI cs.CV cs.IR 67%

Jina CLIP: Your CLIP Model Is Also Your Text Retriever

Andreas Koukounas, Georgios Mastrapas, Michael Günther, Bo Wang, Scott Martens, Isabelle Mohr, Saba Sturua, Mohammad Kalim Akram, Joan Fontanals Martínez, Saahil Ognawala, Susana Guzman, Maximilian Werk, Nan Wang, Han Xiao

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 4 pages, MFM-EAI@ICML2024

2308.07706 2024-06-21 cs.CV cs.AI cs.CL cs.LG 67%

Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models

Kanchan Poudel, Manish Dhakal, Prasiddha Bhandari, Rabin Adhikari, Safal Thapaliya, Bishesh Khanal

专题命中图文多模态：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Medical Imaging with Deep Learning (MIDL) 2024 (Oral)

2406.12742 2024-06-19 cs.CV cs.AI cs.CL 67%

Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning

Bingchen Zhao, Yongshuo Zong, Letian Zhang, Timothy Hospedales

专题命中图文多模态：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments First three authors contributed equally. Dataset: https://huggingface.co/datasets/VLLMs/MIRB

2406.11069 2024-06-18 cs.CV cs.AI cs.CL 67%

WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences

Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, Bill Yuchen Lin

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments link: https://hf.co/spaces/WildVision/vision-arena

2406.03071 2024-06-06 cs.CV cs.AI cs.MM 67%

Exploiting LMM-based knowledge for image classification tasks

Maria Tzelepi, Vasileios Mezaris

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted for publication, 25th Int. Conf. on Engineering Applications of Neural Networks (EANN/EAAAI 2024), Corfu, Greece, June 2024. This is the "submitted manuscript"

2405.18258 2024-05-29 cs.CV cs.AI cs.CL 67%

Text-only Synthesis for Image Captioning

Qing Zhou, Junlin Huang, Qiang Li, Junyu Gao, Qi Wang

专题命中图文多模态：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

2306.01879 2024-05-16 cs.CV cs.AI cs.CL 67%

Revisiting the Role of Language Priors in Vision-Language Models

Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, Deva Ramanan

专题命中图文多模态：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Published at ICML 2024. Website: https://linzhiqiu.github.io/papers/visual_gpt_score/

2402.01345 2024-05-16 cs.CV cs.AI cs.CL cs.LG 67%

Skip \n: A Simple Method to Reduce Hallucination in Large Vision-Language Models

Zongbo Han, Zechen Bai, Haiyang Mei, Qianli Xu, Changqing Zhang, Mike Zheng Shou

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

2312.07661 2024-05-08 cs.CV cs.CL cs.LG cs.MM 67%

CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor

Shuyang Sun, Runjia Li, Philip Torr, Xiuye Gu, Siyang Li

专题命中图文多模态：image-text(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments To appear in CVPR 2024. Project page: https://torrvision.com/clip_as_rnn/

2404.09690 2024-04-16 cs.CV cs.AI cs.CL cs.LG 67%

Harnessing GPT-4V(ision) for Insurance: A Preliminary Exploration

Chenwei Lin, Hanjia Lyu, Jiebo Luo, Xian Xu

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

2403.19837 2024-04-12 cs.LG cs.AI cs.CL cs.CV cs.LO 67%

Concept-based Analysis of Neural Networks via Vision-Language Models

Ravi Mangal, Nina Narodytska, Divya Gopinath, Boyue Caroline Hu, Anirban Roy, Susmit Jha, Corina Pasareanu

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

2404.02904 2024-04-04 cs.CV cs.AI cs.CL cs.LG 67%

ALOHa: A New Measure for Hallucination in Captioning Models

Suzanne Petryk, David M. Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph E. Gonzalez, Trevor Darrell

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments To appear at NAACL 2024

2402.17128 2024-04-04 cs.CV cs.AI cs.CL cs.LG 67%

OSCaR: Object State Captioning and State Change Representation

Nguyen Nguyen, Jing Bi, Ali Vosoughi, Yapeng Tian, Pooyan Fazli, Chenliang Xu

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NAACL 2024

2403.07839 2024-03-13 cs.CV cs.AI cs.MM 67%

MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric

Haokun Lin, Haoli Bai, Zhili Liu, Lu Hou, Muyi Sun, Linqi Song, Ying Wei, Zhenan Sun

专题命中图文多模态：cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 18 pages, 8 figures, Published in CVPR2024

Journal ref In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

2403.02325 2024-03-05 cs.CV cs.AI cs.CL cs.LG 67%

Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training

David Wan, Jaemin Cho, Elias Stengel-Eskin, Mohit Bansal

专题命中图文多模态：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project website: https://contrastive-region-guidance.github.io/

2402.12728 2024-03-05 cs.CV cs.AI cs.CL cs.IR cs.LG 67%

Modality-Aware Integration with Large Language Models for Knowledge-based Visual Question Answering

Junnan Dong, Qinggang Zhang, Huachi Zhou, Daochen Zha, Pai Zheng, Xiao Huang

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 8 pages,3 figures and 1 page appendix; The processed graphs and codes will be avalibale

2401.17221 2024-01-31 cs.CV cs.AI cs.CL cs.LG 67%

MouSi: Poly-Visual-Expert Vision-Language Models

Xiaoran Fan, Tao Ji, Changhao Jiang, Shuo Li, Senjie Jin, Sirui Song, Junke Wang, Boyang Hong, Lu Chen, Guodong Zheng, Ming Zhang, Caishuang Huang, Rui Zheng, Zhiheng Xi, Yuhao Zhou, Shihan Dou, Junjie Ye, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang

专题命中图文多模态：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

2310.02998 2024-01-29 cs.CV cs.AI cs.CL cs.LG 67%

ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models

Yi-Lin Sung, Jaehong Yoon, Mohit Bansal

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ICLR 2024 (project page: https://ecoflap.github.io/)

2311.04498 2023-12-19 cs.CV cs.AI cs.CL 67%

NExT-Chat: An LMM for Chat, Detection and Segmentation

Ao Zhang, Yuan Yao, Wei Ji, Zhiyuan Liu, Tat-Seng Chua

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Technical Report (https://next-chatv.github.io/)

2312.03818 2023-12-15 cs.CV cs.AI cs.CL cs.LG 67%

Alpha-CLIP: A CLIP Model Focusing on Wherever You Want

Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, Jiaqi Wang

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments project page: https://aleafy.github.io/alpha-clip code: https://github.com/SunzeY/AlphaCLIP

2304.08485 2023-12-14 cs.CV cs.AI cs.CL cs.LG 67%

Visual Instruction Tuning

Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NeurIPS 2023 Oral; project page: https://llava-vl.github.io/

2312.04837 2023-12-13 cs.AI cs.CL cs.CV 67%

Localized Symbolic Knowledge Distillation for Visual Commonsense Models

Jae Sung Park, Jack Hessel, Khyathi Raghavi Chandu, Paul Pu Liang, Ximing Lu, Peter West, Youngjae Yu, Qiuyuan Huang, Jianfeng Gao, Ali Farhadi, Yejin Choi

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Neurips 2023

2311.10933 2023-11-21 cs.AI cs.CL cs.CV cs.HC 67%

Representing visual classification as a linear combination of words

Shobhit Agarwal, Yevgeniy R. Semenov, William Lotter

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments To be published in the Proceedings of the 3rd Machine Learning for Health symposium, Proceedings of Machine Learning Research (PMLR)

2305.17718 2023-11-17 cs.CV cs.AI cs.CL 67%

FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions

Noam Rotstein, David Bensaid, Shaked Brody, Roy Ganz, Ron Kimmel

专题命中图文多模态：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

2305.16934 2023-10-31 cs.CV cs.CL cs.CR cs.LG cs.MM 67%

On Evaluating Adversarial Robustness of Large Vision-Language Models

Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Cheung, Min Lin

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments NeurIPS 2023

2305.10355 2023-10-27 cs.CV cs.CL cs.MM 67%

Evaluating Object Hallucination in Large Vision-Language Models

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, Ji-Rong Wen

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted to EMNLP 2023

2308.14179 2023-08-29 cs.CL cs.AI cs.CV 67%

Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP

Vedant Palit, Rohan Pandey, Aryaman Arora, Paul Pu Liang

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Final version for 5th Workshop on Closing the Loop Between Vision and Language (CLVL) @ ICCV 2023. 4 pages, 5 figures

2305.01788 2023-07-25 cs.CL cs.AI cs.CV 67%

Vision Meets Definitions: Unsupervised Visual Word Sense Disambiguation Incorporating Gloss Information

Sunjae Kwon, Rishabh Garodia, Minhwa Lee, Zhichao Yang, Hong Yu

专题命中图文多模态：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL 2023, https://aclanthology.org/2023.acl-long.88

2306.00228 2023-06-02 cs.CV cs.AI cs.CL 67%

Using Visual Cropping to Enhance Fine-Detail Question Answering of BLIP-Family Models

Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, Filip Ilievski

专题命中图文多模态：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 16 pages, 5 figures, 7 tables

1. 图文多模态 4657 篇

PathAlign: A vision-language model for whole slide images in histopathology

Jina CLIP: Your CLIP Model Is Also Your Text Retriever

Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models

Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning

WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences

Exploiting LMM-based knowledge for image classification tasks

Text-only Synthesis for Image Captioning

Revisiting the Role of Language Priors in Vision-Language Models

Skip \n: A Simple Method to Reduce Hallucination in Large Vision-Language Models

CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor

Harnessing GPT-4V(ision) for Insurance: A Preliminary Exploration

Concept-based Analysis of Neural Networks via Vision-Language Models

ALOHa: A New Measure for Hallucination in Captioning Models

OSCaR: Object State Captioning and State Change Representation

MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric

Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training

Modality-Aware Integration with Large Language Models for Knowledge-based Visual Question Answering

MouSi: Poly-Visual-Expert Vision-Language Models

ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models

NExT-Chat: An LMM for Chat, Detection and Segmentation

Alpha-CLIP: A CLIP Model Focusing on Wherever You Want

Visual Instruction Tuning

Localized Symbolic Knowledge Distillation for Visual Commonsense Models

Representing visual classification as a linear combination of words

FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions

On Evaluating Adversarial Robustness of Large Vision-Language Models

Evaluating Object Hallucination in Large Vision-Language Models

Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP

Vision Meets Definitions: Unsupervised Visual Word Sense Disambiguation Incorporating Gloss Information

Using Visual Cropping to Enhance Fine-Detail Question Answering of BLIP-Family Models