超越标准：性与生殖健康LLM基准测试中的文化错位
Beyond the Rubric: Cultural Misalignment in LLM Benchmarks for Sexual and Reproductive Health
AI总结：
本文指出现有LLM基准测试在性与生殖健康领域存在文化偏见，提出需开发适应不同文化需求的评估框架。
摘要：
大型语言模型（LLMs）被定位为有潜力扩大全球南方获取健康信息的途径，但其评估仍然严重依赖于围绕西方规范设计的基准测试。我们呈现了与印度一个未被满足社区中的性与生殖健康（SRH）聊天机器人进行的初步基准测试的见解。我们使用OpenAI设计的对话健康模型基准测试HealthBench进行评估。我们从数据集中提取了637个SRH查询，并在330个单轮对话上进行评估。响应通过HealthBench的基于rubric的自动评分器进行评估，该评分器对响应评分一贯偏低。然而，由受过训练的标注员和公共卫生专家进行的定性分析发现，许多响应实际上是文化上恰当且医学上准确的。我们强调了反复出现的问题，特别是西方偏见，例如法律框架和规范（如公共场所哺乳），饮食假设（如怀孕期间鱼类是否安全食用），以及成本（如保险模型）。我们的发现表明，当前基准测试在捕捉为不同文化和医疗环境构建的系统有效性方面存在局限性。我们呼吁开发能够适应文化需求的评估框架，以满足不同群体的需求。
英文摘要：
Large Language Models (LLMs) have been positioned as having the potential to expand access to health information in the Global South, yet their evaluation remains heavily dependent on benchmarks designed around Western norms. We present insights from a preliminary benchmarking exercise with a chatbot for sexual and reproductive health (SRH) for an underserved community in India. We evaluated using HealthBench, a benchmark for conversational health models by OpenAI. We extracted 637 SRH queries from the dataset and evaluated on the 330 single-turn conversations. Responses were evaluated using HealthBench's rubric-based automated grader, which rated responses consistently low. However, qualitative analysis by trained annotators and public health experts revealed that many responses were actually culturally appropriate and medically accurate. We highlight recurring issues, particularly a Western bias, such as for legal framing and norms (e.g., breastfeeding in public), diet assumptions (e.g., fish safe to eat during pregnancy), and costs (e.g., insurance models). Our findings demonstrate the limitations of current benchmarks in capturing the effectiveness of systems built for different cultural and healthcare contexts. We argue for the development of culturally adaptive evaluation frameworks that meet quality standards while recognizing needs of diverse populations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
