arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.09201cs.CLcs.AI
收藏

用于热带病与传染病分类的大语言模型情境化评估

Contextual Evaluation of Large Language Models for Classifying Tropical and Infectious Diseases

Google Research（谷歌研究院）；Google DeepMind（谷歌DeepMind）；University of Michigan（密歇根大学）；Gear Health（吉尔健康）；Ministry of Health, Sierra Leone（塞拉利昂卫生部）；Nigerian Center for Disease Control（尼日利亚疾病控制中心）；Kenyatta University（肯雅塔大学）
Mercy Asiedu, Nenad Tomasev, Chintan Ghate, Tiya Tiyasirichokchai, Awa Dieng, Oluwatosin Akande, Geoffrey Siwo, Steve Adudans, Sylvanus Aitkins, Odianosen Ehiak… 展开作者

Mercy Asiedu, Nenad Tomasev, Chintan Ghate, Tiya Tiyasirichokchai, Awa Dieng, Oluwatosin Akande, Geoffrey Siwo, Steve Adudans, Sylvanus Aitkins, Odianosen Ehiakhamen, Eric Ndombi, Katherine Heller

备注：
Accepted at 2 NeurIPS 2024 workshops: Generative AI for Health Workshop and Workshop on Advancements In Medical Foundation Models: Explainability, Robustness, Security, and Beyond
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

该研究针对热带病与传染病领域LLM专项研究不足的问题，扩展TRINDs数据集生成超11000条提示，对比评估不同LLM及人类专家的表现，验证情境信息的优化作用，并开发TRINDs-LM研究工具原型。

摘要：

尽管大语言模型（LLM）在医疗问答领域已展现出应用潜力，但针对热带病与传染病的专项研究仍较为有限。我们基于开源的热带与传染病（TRINDs）数据集开展工作，对其进行扩展，加入人口统计学特征、语义化临床与消费者端增强数据，最终生成超过11000条提示词。我们在这些数据上评估LLM的表现，对比通用型与医疗专用LLM的性能，同时将LLM的输出结果与人类专家的判断进行比较。通过系统性实验，我们证实了人口统计信息、地理位置、性别、风险因素等情境信息对于优化LLM响应的作用。最后，我们开发了TRINDs-LM原型，这是一款研究工具，可提供交互平台以探索情境如何影响LLM在健康领域的输出。

英文摘要：

While large language models (LLMs) have shown promise for medical question answering, there is limited work focused on tropical and infectious disease-specific exploration. We build on an opensource tropical and infectious diseases (TRINDs) dataset, expanding it to include demographic and semantic clinical and consumer augmentations yielding 11000+ prompts. We evaluate LLM performance on these, comparing generalist and medical LLMs, as well as LLM outcomes to human experts. We demonstrate through systematic experimentation, the benefit of contextual information such as demographics, location, gender, risk factors for optimal LLM response. Finally we develop a prototype of TRINDs-LM, a research tool that provides a playground to navigate how context impacts LLM outputs for health.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑