CCBENCH：通过健康查询使用隐式信号规范评估大语言模型的文化能力
CCBENCH: Assessing LLM Cultural Competence via Implicitly Signaled Norms using Health Queries
AI总结：
该研究引入CCBENCH框架，以健康领域为案例创建CCBENCH-Health评估大语言模型文化能力。通过60个角色、3120次互动对五个领先模型进行基准测试，发现模型文化恰当回应比例低，提示关注文化线索能适度提升性能，还揭示了模型与文化线索互动的一些特点。
摘要：
为了与用户公平互动且无刻板印象，人工智能模型必须展现文化能力，即推断并适应用户隐式信号文化价值观的能力。我们引入CCBENCH框架来评估大语言模型的文化能力，将文化视为规范遵守状态的连续体而非文化归属的二元状态。以健康领域为例创建了CCBENCH-Health，包含60个跨六种文化展现不同规范遵守状态的角色，每个角色进行18次真实对话，基于52个真实医疗问题评估，产生3120次独特互动。对五个领先模型的基准测试表明，即使最佳模型在文化上恰当回应的时间也仅为20%-30%。当明确提示关注对话历史中的文化相关线索时，性能平均适度提高3%-5%。发现模型在角色避免文化规范而非遵循时表现最佳，揭示了持续的不对称性，表明模型更倾向于与固有偏差一致而非适应文化线索，在阿富汗背景下尤其明显。最后发现模型有时更易适应隐式文化对话风格而非明确陈述的文化实践，且因文化而异。
英文摘要：
To interact with users fairly and without stereotyping, AI models must display cultural competency, i.e., the ability to infer and adapt to a user's implicitly signaled cultural values, rather than relying on static demographic traits. We introduce CCBENCH, a framework for evaluating cultural competency in large language models (LLMs), treating culture as a continuum of norm adherence states rather than as a binary state of cultural belongingness. As a case study on health, we create CCBENCH-Health, which includes 60 theoretically grounded personas exhibiting varied norm-adherence states across six cultures, each engaging in 18 realistic dialogues. Each persona is evaluated on 52 authentic healthcare questions drawn from real user forums, yielding 3,120 unique interactions. Benchmarking five leading models reveals that even the best achieve culturally appropriate responses only 20-30% of the time. When explicitly prompted to focus on culturally relevant cues from the conversational history (CoT), performance improves modestly by 3-5% on average. We find that models perform best when personas avoid cultural norms rather than follow them, revealing a persistent asymmetry, suggesting a preference in the models to align with built-in biases than adapt to cultural cues. This is especially observed in the Afghan context (Avg: 8.8%), where cultural cues rarely yield appropriate health advice. Finally, we find that models sometimes adapt more readily to implicit, cultural conversational styles than to explicitly stated cultural practices, though this varies across cultures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
