健康AI评估中的有效性缺口:基准组成的横断面分析
The Validity Gap in Health AI Evaluation: A Cross-Sectional Analysis of Benchmark Composition
AI总结:
研究指出当前健康AI基准测试在临床需求与实际应用间存在有效性缺口,缺乏真实临床数据和脆弱群体代表,需建立标准化查询分析以提升评估准确性。
摘要:
背景:临床试验依赖透明的纳入标准以确保可推广性。相比之下,验证健康相关大语言模型(LLMs)的基准测试很少描述它们所包含的'患者'或'查询'人群。没有明确定义的组成,聚合性能指标可能误导模型在临床中的应用准备情况。方法:我们利用LLMs作为自动化编码工具,分析了六个公开基准中的18,707个消费者健康查询,应用标准化的16字段分类法来描述上下文、主题和意图。结果:我们识别出结构性的'有效性缺口'。尽管基准从静态检索发展到互动对话,但临床组成仍与现实需求不匹配。虽然42%的语料引用客观数据,但偏向于以健康为中心的可穿戴信号(17.7%);复杂诊断输入仍罕见,包括实验室值(5.2%)、影像(3.8%)和原始医疗记录(0.6%)。安全关键场景几乎缺失:自杀/自残查询占语料的<0.7%,慢性病管理仅5.5%。基准也忽视了脆弱人群(儿童/老年人<11%)和全球健康需求。结论:评估基准仍与现实临床需求不匹配,缺乏原始临床数据、充分的脆弱人群代表和长期慢性护理场景。该领域必须采用标准化查询分析——类似于临床试验报告——以使评估与临床实践的全部复杂性对齐。
英文摘要:
Background: Clinical trials rely on transparent inclusion criteria to ensure generalizability. In contrast, benchmarks validating health-related large language models (LLMs) rarely characterize the "patient" or "query" populations they contain. Without defined composition, aggregate performance metrics may misrepresent model readiness for clinical use. Methods: We analyzed 18,707 consumer health queries across six public benchmarks using LLMs as automated coding instruments to apply a standardized 16-field taxonomy profiling context, topic, and intent. Results: We identified a structural "validity gap." While benchmarks have evolved from static retrieval to interactive dialogue, clinical composition remains misaligned with real-world needs. Although 42% of the corpus referenced objective data, this was polarized toward wellness-focused wearable signals (17.7%); complex diagnostic inputs remained rare, including laboratory values (5.2%), imaging (3.8%), and raw medical records (0.6%). Safety-critical scenarios were effectively absent: suicide/self-harm queries comprised <0.7% of the corpus and chronic disease management only 5.5%. Benchmarks also neglected vulnerable populations (pediatrics/older adults <11%) and global health needs. Conclusions: Evaluation benchmarks remain misaligned with real-world clinical needs, lacking raw clinical artifacts, adequate representation of vulnerable populations, and longitudinal chronic care scenarios. The field must adopt standardized query profiling--analogous to clinical trial reporting--to align evaluation with the full complexity of clinical practice.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。