arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.06569cs.CLcs.AI
收藏

公平多模态大语言模型的社会偏见消除

Social Debiasing for Fair Multi-modal LLMs

National University of Singapore(新加坡国立大学);University of Warwick(华威大学);Ant Group(蚂蚁集团);Shandong University(山东大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))

Harry Cheng, Yangyang Guo, Qingpei Guo, Ming Yang, Tian Gan, Weili Guan, Liqiang Nie

AI总结:

针对多模态大语言模型(MLLMs)的社会偏见问题,本文构建含18种平衡社会概念的综合反事实数据集(CMSC),提出结合偏见感知采样与损失重缩放的反刻板印象去偏(CSD)策略,在不降低通用性能的前提下有效减少偏见。

摘要:

多模态大语言模型(MLLMs)极大推动了研究领域发展,具备强大的视觉-语言理解能力。但这些模型常从训练数据中继承根深蒂固的社会偏见,在种族、性别等属性上产生不当响应。本文通过两点解决MLLMs的社会偏见问题:一是引入包含多社会概念的综合反事实数据集(CMSC),提供18种多样且平衡的社会概念以补充现有数据集;二是提出反刻板印象去偏(CSD)策略,利用普遍刻板印象的对立面缓解MLLMs的社会偏见。CSD结合新颖的偏见感知数据采样方法与损失重缩放方法,有效减少模型偏见。我们对四种主流MLLM架构进行大量实验,结果表明CMSC数据集的优势及CSD策略相比现有竞争方法在减少社会偏见上的优势,且不影响通用多模态推理基准的整体性能。

英文摘要:

Multi-modal Large Language Models (MLLMs) have dramatically advanced the research field and delivered powerful vision-language understanding capabilities. However, these models often inherit deep-rooted social biases from their training data, leading to uncomfortable responses with respect to attributes such as race and gender. This paper addresses the issue of social biases in MLLMs by i) introducing a comprehensive counterfactual dataset with multiple social concepts (CMSC), which complements existing datasets by providing 18 diverse and balanced social concepts; and ii) proposing a counter-stereotype debiasing (CSD) strategy that mitigates social biases in MLLMs by leveraging the opposites of prevalent stereotypes. CSD incorporates both a novel bias-aware data sampling method and a loss rescaling method, enabling the model to effectively reduce biases. We conduct extensive experiments with four prevalent MLLM architectures. The results demonstrate the advantage of the CMSC dataset and the edge of CSD strategy in reducing social biases compared to existing competing methods, without compromising the overall performance on general multi-modal reasoning benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑