偏见超越人口统计：通过反事实视觉问答探测黑盒大视觉-语言模型的决策边界
Bias Beyond Demographics: Probing Decision Boundaries in Black-Box LVLMs via Counterfactual VQA
AI总结：
本文通过反事实VQA基准探测黑盒LVLMs的决策边界，揭示非人口属性对决策的更大影响，并展示人类规范验证示例对提升模型响应一致性和公平性的作用。
摘要：
近年来，大视觉-语言模型（LVLMs）的进步加剧了对公平性的关注，但现有评估仍局限于人口属性，并且往往将公平性与拒绝行为混为一谈。本文通过引入一个反事实VQA基准，扩大公平性研究的范围，探测封闭源LVLMs在受控上下文偏移下的决策边界。每对图像仅在单一视觉属性上不同，该属性已被验证与问题无关，从而实现无地面真实值和拒绝意识的推理稳定性分析。全面的实验表明，非人口属性，如环境背景或社会行为，比人口属性更强烈地扭曲LVLM的决策。此外，基于指令的去偏置方法显示效果有限，甚至可能放大这些不对称性，而暴露于我们基准中少量经过人类规范验证的例子，能促使更一致和平衡的响应，突显其不仅作为评估框架，也作为理解和改进模型行为的手段的潜力。这些结果为审计黑盒LVLMs的上下文偏见提供了实用基础，并有助于更透明和公平的多模态推理。
英文摘要：
Recent advances in large vision-language models (LVLMs) have amplified concerns about fairness, yet existing evaluations remain confined to demographic attributes and often conflate fairness with refusal behavior. This paper broadens the scope of fairness by introducing a counterfactual VQA benchmark that probes the decision boundaries of closed-source LVLMs under controlled context shifts. Each image pair differs in a single visual attribute that has been validated as irrelevant to the question, enabling ground-truth-free and refusal-aware analysis of reasoning stability. Comprehensive experiments reveal that non-demographic attributes, such as environmental context or social behavior, distort LVLM decision-making more strongly than demographic ones. Moreover, instruction-based debiasing shows limited effectiveness and can even amplify these asymmetries, whereas exposure to a small number of human norm validated examples from our benchmark encourages more consistent and balanced responses, highlighting its potential not only as an evaluative framework but also as a means for understanding and improving model behavior. Together, these results provide a practial basis for auditing contextual biases even in black-box LVLMs and contribute to more transparent and equitable multimodal reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
