视觉大模型 / VLM

Low-hallucination Synthetic Captions for Large-Scale Vision-Language Model Pre-training

Xinsong Zhang, Yarong Zeng, Xinting Huang, Hu Hu, Runquan Xie, Han Hu, Zhanhui Kang

机构 * Tencent（腾讯） 纠错

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

Data-Driven Calibration of Prediction Sets in Large Vision-Language Models Based on Inductive Conformal Prediction

Yuanchang Ye, Weiyan Wen

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；visual question answering(abstract)；分类 cs.AI、cs.LG

Comments Accepted by ICIPCA 2025

Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks

Han Wang, Gang Wang, Huan Zhang

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） 纠错

专题命中 幻觉与鲁棒性 ：vision language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

MLLM-as-a-Judge for Image Safety without Human Labeling

Zhenting Wang, Shuming Hu, Shiyu Zhao, Xiaowen Lin, Felix Juefei-Xu, Zhuowei Li, Ligong Han, Harihar Subramanyam, Li Chen, Jianfa Chen, Nan Jiang, Lingjuan Lyu, Shiqing Ma, Dimitris N. Metaxas, Ankit Jain

专题命中 幻觉与鲁棒性 ：MLLM(title,abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.LG

AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization

Chaohu Liu, Tianyi Gui, Yu Liu, Linli Xu

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

Enhancing Subsequent Video Retrieval via Vision-Language Models (VLMs)

Yicheng Duan, Xi Huang, Duo Chen

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

TAIJI: Textual Anchoring for Immunizing Jailbreak Images in Vision Language Models

Xiangyu Yin, Yi Qi, Jinwei Hu, Zhen Chen, Yi Dong, Xingyu Zhao, Xiaowei Huang, Wenjie Ruan

专题命中 幻觉与鲁棒性 ：vision language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions

Hadi Amini, Md Jueal Mia, Yasaman Saadati, Ahmed Imteaj, Seyedsina Nabavirazavi, Urmish Thakker, Md Zarif Hossain, Awal Ahmed Fime, S. S. Iyengar

专题命中 幻觉与鲁棒性 ：multimodal large language model(title)；vision language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.LG

MBQ: Modality-Balanced Quantization for Large Vision-Language Models

Shiyao Li, Yingchun Hu, Xuefei Ning, Xihui Liu, Ke Hong, Xiaotao Jia, Xiuhong Li, Yaqi Yan, Pei Ran, Guohao Dai, Shengen Yan, Huazhong Yang, Yu Wang

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoders

Qing Li, Jiahui Geng, Derui Zhu, Fengyu Cai, Chenyang Lyu, Fakhri Karray

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

Comments More comparative experiments are needed

Towards Statistical Factuality Guarantee for Large Vision-Language Models

Zhuohang Li, Chao Yan, Nicholas J. Jackson, Wendi Cui, Bo Li, Jiaxin Zhang, Bradley A. Malin

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.LG

Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images

Yubo Wang, Jianting Tang, Chaohu Liu, Linli Xu

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；visual question answering(abstract)；分类 cs.AI、cs.LG

Comments Accepted to ICLR 2025

EigenShield: Causal Subspace Filtering via Random Matrix Theory for Adversarially Robust Vision-Language Models

Nastaran Darabi, Devashri Naik, Sina Tayebati, Dinithi Jayasuriya, Ranganath Krishnan, Amit Ranjan Trivedi

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Learning Conformal Abstention Policies for Adaptive Risk Management in Large Language and Vision-Language Models

Sina Tayebati, Divake Kumar, Nastaran Darabi, Dinithi Jayasuriya, Ranganath Krishnan, Amit Ranjan Trivedi

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.AI、cs.LG

ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time

Yi Ding, Bolian Li, Ruqi Zhang

专题命中 幻觉与鲁棒性 ：vision language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Comments 29pages

Advancing the Understanding and Evaluation of AR-Generated Scenes: When Vision-Language Models Shine and Stumble

Lin Duan, Yanming Xiu, Maria Gorlatova

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments 6 pages

UniGuard: Towards Universal Safety Guardrails for Jailbreak Attacks on Multimodal Large Language Models

Sejoon Oh, Yiqiao Jin, Megha Sharma, Donghyun Kim, Eric Ma, Gaurav Verma, Srijan Kumar

专题命中 幻觉与鲁棒性 ：multimodal large language model(title,abstract)；LLaVA(abstract)；分类 cs.AI、cs.LG

Comments 14 pages

Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving

Lu Wang, Tianyuan Zhang, Yang Qu, Siyuan Liang, Yuwei Chen, Aishan Liu, Xianglong Liu, Dacheng Tao

专题命中 幻觉与鲁棒性 ：vision language model(title)；vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Read Like a Radiologist: Efficient Vision-Language Model for 3D Medical Imaging Interpretation

Changsun Lee, Sangjoon Park, Cheong-Il Shin, Woo Hee Choi, Hyun Jeong Park, Jeong Eun Lee, Jong Chul Ye

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

More Distinctively Black and Feminine Faces Lead to Increased Stereotyping in Vision-Language Models

Messi H. J. Lee, Jacob M. Montgomery, Calvin K. Lai

专题命中 幻觉与鲁棒性 ：vision-language model(title)；vision language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments This submission is being withdrawn to address concerns related to the terms of use of a database utilized in the research. We aim to ensure full compliance with all data usage agreements before proceeding with publication

ViSTa Dataset: Do vision-language models understand sequential tasks?

Evžen Wybitul, Evan Ryan Gunter, Mikhail Seleznyov, David Lindner

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization

Yuhan Fu, Ruobing Xie, Xingwu Sun, Zhanhui Kang, Xirong Li

专题命中 幻觉与鲁棒性 ：multimodal large language model(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

VHELM: A Holistic Evaluation of Vision Language Models

Tony Lee, Haoqin Tu, Chi Heem Wong, Wenhao Zheng, Yiyang Zhou, Yifan Mai, Josselin Somerville Roberts, Michihiro Yasunaga, Huaxiu Yao, Cihang Xie, Percy Liang

专题命中 幻觉与鲁棒性 ：vision language model(title,abstract)；vision-language model(abstract)；分类 cs.CV、cs.AI

Comments NeurIPS 2024. First three authors contributed equally

Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks

Md Zarif Hossain, Ahmed Imteaj

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation

Xiaoye Qu, Qiyuan Chen, Wei Wei, Jishuo Sun, Jianfeng Dong

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Prompts

Yi Liu, Chengjun Cai, Xiaoli Zhang, Xingliang Yuan, Cong Wang

专题命中 幻觉与鲁棒性 ：vision language model(title,abstract)；VLM(abstract)；分类 cs.AI、cs.LG

Comments To be published in ACM MM 2024

GenderBias-\emph{VL}: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing

Yisong Xiao, Aishan Liu, QianJia Cheng, Zhenfei Yin, Siyuan Liang, Jiapeng Li, Jing Shao, Xianglong Liu, Dacheng Tao

专题命中 幻觉与鲁棒性 ：vision language model(title)；vision-language model(abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

Comments 9 pages, 4 figures

Uncovering Bias in Large Vision-Language Models with Counterfactuals

Phillip Howard, Anahita Bhiwandiwalla, Kathleen C. Fraser, Svetlana Kiritchenko

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Accepted to the CVPR 2024 Responsible Generative AI (ReGenAI) Workshop

Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Models

Tobias Groot, Matias Valdenegro-Toro

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Comments 8 pages, with appendix. To appear in TrustNLP workshop @ NAACL 2024

MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning

Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, Baobao Chang

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ICLR2024