FairLLaVA: 大规模视觉-语言助手中的公平性感知参数高效微调
FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants
AI总结：
FairLLaVA通过减少目标属性间的互信息，实现视觉指令微调中的公平性改进，提升医疗影像生成的公平性和自然语言生成质量。
摘要：
尽管在图像条件生成中表现强大，多模态大语言模型（MLLMs）在不同人口群体间表现不均，凸显公平性风险。在安全关键的临床环境中，此类差异可能产生不平等的诊断叙述并削弱人工智能辅助决策的信任度。尽管在视觉-only和语言-only模型中已广泛研究公平性，但其对MLLMs的影响仍鲜有探索。为解决这些偏见，我们引入FairLLaVA，一种参数高效的微调方法，通过在视觉指令微调中减少群体差异而不影响整体性能。通过最小化目标属性间的互信息，FairLLaVA正则化模型表示以实现人口不变性。该方法可作为轻量级插件整合，维持低秩适配器微调的效率，并提供架构无关的公平视觉指令遵循方法。在大规模胸部X光报告生成和皮肤镜视觉问答基准测试中，广泛实验表明FairLLaVA一致减少了群体间差异，同时提高了公平性尺度的临床性能和自然语言生成质量。代码可在https://github.com/bhosalems/FairLLaVA访问。
英文摘要：
While powerful in image-conditioned generation, multimodal large language models (MLLMs) can display uneven performance across demographic groups, highlighting fairness risks. In safety-critical clinical settings, such disparities risk producing unequal diagnostic narratives and eroding trust in AI-assisted decision-making. While fairness has been studied extensively in vision-only and language-only models, its impact on MLLMs remains largely underexplored. To address these biases, we introduce FairLLaVA, a parameter-efficient fine-tuning method that mitigates group disparities in visual instruction tuning without compromising overall performance. By minimizing the mutual information between target attributes, FairLLaVA regularizes the model's representations to be demographic-invariant. The method can be incorporated as a lightweight plug-in, maintaining efficiency with low-rank adapter fine-tuning, and provides an architecture-agnostic approach to fair visual instruction following. Extensive experiments on large-scale chest radiology report generation and dermoscopy visual question answering benchmarks show that FairLLaVA consistently reduces inter-group disparities while improving both equity-scaled clinical performance and natural language generation quality across diverse medical imaging modalities. Code can be accessed at https://github.com/bhosalems/FairLLaVA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
