视觉大模型 / VLM

Stop learning it all to mitigate visual hallucination, Focus on the hallucination target

Dokyoon Yoon, Youngsook Song, Woomyong Park

机构 * SIONIC AI 纠错

专题命中 幻觉与鲁棒性 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted to CVPR 2025

Journal ref Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

Self-interpreting Adversarial Images

Tingwei Zhang, Collin Zhang, John X. Morris, Eugene Bagdasarian, Vitaly Shmatikov

机构 * Cornell Tech（康奈尔科技） ； University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校） 纠错

专题命中 幻觉与鲁棒性 ：visual language model(abstract)；分类 cs.AI、cs.LG

Comments in USENIX Security 2025

Benchmark Granularity and Model Robustness for Image-Text Retrieval

Mariya Hendriksen, Shuo Zhang, Ridho Reinanda, Mohamed Yahya, Edgar Meij, Maarten de Rijke

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments accepted at SIGIR 2025

Vid-SME: Membership Inference Attacks against Large Video Understanding Models

Qi Li, Runpeng Yu, Xinchao Wang

机构 * National University of Singapore（新加坡国立大学） 纠错

专题命中 幻觉与鲁棒性 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs

Wenxuan Wang, Xiaoyuan Liu, Kuiyi Gao, Jen-tse Huang, Youliang Yuan, Pinjia He, Shuai Wang, Zhaopeng Tu

机构 * Renmin University of China（中国人民大学） ； Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）） ； Chinese University of Hong Kong（香港中文大学） ； Johns Hopkins University（约翰霍普金斯大学） ； Hong Kong University of Science and Technology（香港科技大学） ； Tencent（腾讯） 纠错

专题命中 幻觉与鲁棒性 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ACL 2025

Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward

Zhiyuan Fan, Yumeng Wang, Sandeep Polisetty, Yi R. Fung

机构 * Hong Kong University of Science and Technology（香港理工大学） 纠错

专题命中 幻觉与鲁棒性 ：vision language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted to ACL 2025 Findings

From Hallucinations to Jailbreaks: Rethinking the Vulnerability of Large Foundation Models

Haibo Jin, Peiyan Zhang, Peiran Wang, Man Luo, Haohan Wang

机构 * School of Information Sciences University of Illinois at Urbana-Champaign（信息科学学院伊利诺伊大学厄巴纳-香槟分校） ； Computer Science and Engineering HKUST（计算机科学与工程香港科技大学） ； Computer Science Department University of California, Los Angeles（计算机科学系加州大学洛杉矶分校） ； Research Scientist, Intel Labs（英特尔实验室研究员） ； School of Information Sciences University of Illinois Urbana-Champaign（信息科学学院伊利诺伊大学厄巴纳-香槟分校） 纠错

专题命中 幻觉与鲁棒性 ：LLaVA(abstract)；分类 cs.CV、cs.AI

Benign-to-Toxic Jailbreaking: Inducing Harmful Responses from Harmless Prompts

Hee-Seon Kim, Minbeom Kim, Wonjun Lee, Kihyun Kim, Changick Kim

机构 * Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院） 纠错

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments LVLM, Jailbreak

EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition

Christoph Schuhmann, Robert Kaczmarczyk, Gollam Rabby, Felix Friedrich, Maurice Kraus, Krishna Kalyan, Kourosh Nadi, Huu Nguyen, Kristian Kersting, Sören Auer

机构 * LAION e.V.（LAION研究所） ； Technical University of Munich（慕尼黑技术大学） ； L3S Research Center Leibniz University of Hannover（汉诺威莱布尼茨大学L3S研究中心） ； TU Darmstadt（达姆施塔特技术大学） ； DFKI（德意志联邦防务研究所） ； TIB–Leibniz Information Centre for Science and Technology L3S Research Center Leibniz University of Hannover（莱布尼茨科学与技术信息中心L3S研究中心汉诺威莱布尼茨大学） 纠错

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations

Boxu Chen, Ziwei Zheng, Le Yang, Zeyu Geng, Zhengyu Zhao, Chenhao Lin, Chao Shen

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback

Jiaming Ji, Xinyu Chen, Rui Pan, Conghui Zhang, Han Zhu, Jiahao Li, Donghai Hong, Boyuan Chen, Jiayi Zhou, Kaile Wang, Juntao Dai, Chi-Min Chan, Yida Tang, Sirui Han, Yike Guo, Yaodong Yang

机构 * Peking University（北京大学） ； Hong Kong University of Science and Technology（香港科技大学） 纠错

专题命中 幻觉与鲁棒性 ：multimodal large language model(abstract)；分类 cs.AI、cs.LG

Learning Novel Skills from Language-Generated Demonstrations

Ao-Qun Jin, Tian-Yu Xiang, Xiao-Hu Zhou, Mei-Jiang Gui, Xiao-Liang Xie, Shi-Qi Liu, Shuang-Yi Wang, Yue Cao, Sheng-Bin Duan, Fu-Chao Xie, Zeng-Guang Hou

机构 * Institute of Automation Chinese Academy of Sciences（自动化研究所，中国科学院） 纠错

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.AI、cs.LG

Comments 10 pages, International Conference on Learning Representations (ICLR) 2025 Workshop on Generative Models for Robot Learning (GenBot)

SafeVid: Toward Safety Aligned Video Large Multimodal Models

Yixu Wang, Jiaxin Song, Yifeng Gao, Xin Wang, Yang Yao, Yan Teng, Xingjun Ma, Yingchun Wang, Yu-Gang Jiang

机构 * Fudan University（复旦大学） ； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） 纠错

专题命中 幻觉与鲁棒性 ：LLaVA(abstract)；分类 cs.CV、cs.AI

CUBIC: Concept Embeddings for Unsupervised Bias Identification using VLMs

David Méndez, Gianpaolo Bontempo, Elisa Ficarra, Roberto Confalonieri, Natalia Díaz-Rodríguez

机构 * Dept. of Computer Science and Artificial Intelligence, DaSCI Institute, University of Granada（计算机科学与人工智能系，DaSCI研究所，格拉纳达大学） ； Dept. of Engineering ”Enzo Ferrari”, University of Modena and Reggio Emilia（工程系，摩德纳和雷吉奥艾米利亚大学） ； Dept. of Mathematics ’Tullio Levi-Civita’, University of Padova（数学系，帕多瓦大学） 纠错

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments 8 pages, 3 figures, 5 tables. Accepted at IJCNN 2025; to appear in IEEE Xplore

Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training

Yangyi Chen, Hao Peng, Tong Zhang, Heng Ji

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） 纠错

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments The code will be available at https://github.com/Yangyi-Chen/PRIOR

Don't Deceive Me: Mitigating Gaslighting through Attention Reallocation in LMMs

Pengkun Jiao, Bin Zhu, Jingjing Chen, Chong-Wah Ngo, Yu-Gang Jiang

专题命中 幻觉与鲁棒性 ：LLaVA(abstract)；分类 cs.CV、cs.AI

ReferGPT: Towards Zero-Shot Referring Multi-Object Tracking

Tzoulio Chamiti, Leandro Di Bella, Adrian Munteanu, Nikos Deligiannis

专题命中 幻觉与鲁棒性 ：MLLM(abstract)；分类 cs.CV、cs.AI

Comments Accepted CVPR 2025 Workshop on Distillation of Foundation Models for Autonomous Driving

Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict

Pouya Pezeshkpour, Moin Aminnaseri, Estevam Hruschka

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.CV、cs.AI

ContextIQ: A Multimodal Expert-Based Video Retrieval System for Contextual Advertising

Ashutosh Chaubey, Anoubhav Agarwaal, Sartaki Sinha Roy, Aayush Agrawal, Susmita Ghose

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Published at WACV 2025

Aligning Visual Contrastive learning models via Preference Optimization

Amirabbas Afzali, Borna Khodabandeh, Ali Rasekh, Mahyar JafariNodeh, Sepehr kazemi, Simon Gottschalk

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.CV、cs.LG

CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuning

Hao Yu, Zhuokai Zhao, Shen Yan, Lukasz Korycki, Jianyu Wang, Baosheng He, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Hanchao Yu

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.CV、cs.AI

MF-CLIP: Leveraging CLIP as Surrogate Models for No-box Adversarial Attacks

Jiaming Zhang, Lingyu Qiu, Qi Yi, Yige Li, Jitao Sang, Changsheng Xu, Dit-Yan Yeung

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.AI、cs.LG

good4cir: Generating Detailed Synthetic Captions for Composed Image Retrieval

Pranavi Kolouju, Eric Xing, Robert Pless, Nathan Jacobs, Abby Stylianou

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Classifier-guided CLIP Distillation for Unsupervised Multi-label Classification

Dongseob Kim, Hyunjung Shim

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments CVPR 2025 Accepted

PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training

Cong Chen, Mingyu Liu, Chenchen Jing, Yizhou Zhou, Fengyun Rao, Hao Chen, Bo Zhang, Chunhua Shen

专题命中 幻觉与鲁棒性 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

Octopus: Alleviating Hallucination via Dynamic Contrastive Decoding

Wei Suo, Lijun Zhang, Mengyang Sun, Lin Yuanbo Wu, Peng Wang, Yanning Zhang

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.CV、cs.AI

See It from My Perspective: How Language Affects Cultural Bias in Image Understanding

Amith Ananthram, Elias Stengel-Eskin, Mohit Bansal, Kathleen McKeown

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted at ICLR 2025. 22 pages, 6 figures. Code/models: https://github.com/amith-ananthram/see-it-from-my-perspective

VidLBEval: Benchmarking and Mitigating Language Bias in Video-Involved LVLMs

Yiming Yang, Yangyang Guo, Hui Lu, Yan Wang

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Can Hallucination Correction Improve Video-Language Alignment?

Lingjun Zhao, Mingyang Xie, Paola Cascante-Bonilla, Hal Daumé, Kwonjoon Lee

专题命中 幻觉与鲁棒性 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Magnifier Prompt: Tackling Multimodal Hallucination via Extremely Simple Instructions

Yuhan Fu, Ruobing Xie, Jiazhen Liu, Bangxiang Lan, Xingwu Sun, Zhanhui Kang, Xirong Li

专题命中 幻觉与鲁棒性 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments The proposed method does not work for up-to-date MLLMs.