反事实文化线索降低大语言模型医学问答准确性：标识符效应与情境效应
Counterfactual Cultural Cues Reduce Medical QA Accuracy in LLMs: Identifier vs Context Effects
AI总结：
本研究通过反事实文化线索测试，发现文化相关信息显著降低大语言模型医学问答准确性，尤其在标识符与情境共同存在时影响最大。
摘要：
构建可持续且公平的医疗保健系统需要一种不会在面对非决定性文化信息时改变临床正确诊断的医学语言模型。我们引入了一个反事实基准测试，通过插入与文化相关（i）标识符标记、（ii）情境线索或（iii）其组合，将150个MedQA测试项目扩展为1650种变体，针对三个群体（原住民加拿大人、中东穆斯林、东南亚人）以及一个长度匹配的中性对照组，其中临床医生验证了黄金答案在所有变体中保持不变。我们评估了GPT-5.2、Llama-3.1-8B、DeepSeek-R1和MedGemma（4B/27B）在仅选项和简要解释提示下的表现。在所有模型中，文化线索显著影响准确性（Cochran's Q，p<10^-14），当标识符和情境同时出现时，准确性下降最大（在仅选项提示下下降3-7个百分点），而中性编辑产生的变化较小且非系统性。通过LLM作为评判者的经人类验证的评分表（κ=0.76）显示，超过一半的文化相关解释最终导致错误答案，将文化参照性推理与诊断失败联系起来。我们发布了提示和增强以支持评估和缓解文化诱导的诊断错误。
英文摘要：
Engineering sustainable and equitable healthcare requires medical language models that do not change clinically correct diagnoses when presented with non-decisive cultural information. We introduce a counterfactual benchmark that expands 150 MedQA test items into 1650 variants by inserting culture-related (i) identifier tokens, (ii) contextual cues, or (iii) their combination for three groups (Indigenous Canadian, Middle-Eastern Muslim, Southeast Asian), plus a length-matched neutral control, where a clinician verified that the gold answer remains invariant in all variants. We evaluate GPT-5.2, Llama-3.1-8B, DeepSeek-R1, and MedGemma (4B/27B) under option-only and brief-explanation prompting. Across models, cultural cues significantly affect accuracy (Cochran's Q, $p<10^-14$), with the largest degradation when identifier and context co-occur (up to 3-7 percentage points under option-only prompting), while neutral edits produce smaller, non-systematic changes. A human-validated rubric ($κ=0.76$) applied via an LLM-as-judge shows that more than half of culturally grounded explanations end in an incorrect answer, linking culture-referential reasoning to diagnostic failure. We release prompts and augmentations to support evaluation and mitigation of culturally induced diagnostic errors.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
