arXiv 2505.195622026-01-11cs.AI

FairMedQA: 对大型语言模型在医学问答中的偏见进行基准测试

FairMedQA: Benchmarking Bias in Large Language Models for Medical Question Answering

King’s College London（伦敦国王学院）；University of Electronic Science and Technology of China（电子科技大学）；Sun Yat-sen University（中山大学）；York University（约克大学）；Southern University of Science and Technology（南方科技大学）；University College London（伦敦大学学院）；Tsinghua University（清华大学）

Ying Xiao, Jie Huang, Ruijuan He, Jing Xiao, Mohammad Reza Mousavi, Yepang Liu, Kezhi Li, Zhenpeng Chen, Jie M. Zhang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

FairMedQA基准测试揭示了大型语言模型在医学问答中存在显著偏见，需进一步开发去偏技术和严格验证方法。

摘要：

大型语言模型（LLMs）在医学问答中的表现正接近专家水平，显示出改善公共医疗的潜力。然而，与性别和种族等敏感属性相关的潜在偏见可能带来生命攸关的风险。这些敏感属性对诊断影响的程度仍是一个开放性问题，需要全面的实证研究。此外，即使最新的反事实患者变异（CPV）基准也难以区分不同LLM的偏见水平。为了进一步探索这些动态，我们提出一个新的基准，FairMedQA，并对12个代表性LLM进行了基准测试。FairMedQA包含4,806个反事实问题对，这些问题对来源于801个临床病例片段。我们的结果揭示了在敏感人口群体中准确率差异范围从3到19个百分点。值得注意的是，FairMedQA揭示的偏见至少比最新CPV基准发现的偏见大12个百分点，展示了更优越的基准测试敏感性。我们的结果强调了在LLM安全集成到实际临床决策支持系统之前，需要有针对性的去偏技术以及更严格、身份意识的验证协议的紧迫性。

英文摘要：

Large language models (LLMs) are approaching expert-level performance in medical question answering (QA), demonstrating strong potential to improve public healthcare. However, underlying biases related to sensitive attributes such as sex and race pose life-critical risks. The extent to which such sensitive attributes affect diagnosis remains an open question and requires comprehensive empirical investigation. Additionally, even the latest Counterfactual Patient Variations (CPV) benchmark can hardly distinguish the bias levels of different LLMs. To further explore these dynamics, we propose a new benchmark, FairMedQA, and benchmark 12 representative LLMs. FairMedQA contains 4,806 counterfactual question pairs constructed from 801 clinical vignettes. Our results reveal substantial accuracy disparity ranging from 3 to 19 percentage points across sensitive demographic groups. Notably, FairMedQA exposes biases that are at least 12 percentage points larger than those identified by the latest CPV benchmark, presenting superior benchmarking sensitivity. Our results underscore an urgent need for targeted debiasing techniques and more rigorous, identity-aware validation protocols before LLMs can be safely integrated into practical clinical decision-support systems.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。