上下文对AI健康建议准确性的影响有多大？
How much does context affect the accuracy of AI health advice?
AI总结：
本文研究了上下文对AI健康建议准确性的影响，发现语言、主题和信息源显著影响LLM的健康声明验证性能，强调了多语言和领域特定评估的重要性。
摘要：
大型语言模型（LLMs）越来越多地被用于提供健康建议，但关于其在不同语言、主题和信息来源下准确性变化的证据仍然有限。我们评估了语言和上下文因素如何影响基于AI的健康声明验证的准确性。我们对七种广泛使用的LLMs在两个数据集上进行了评估：（i）1975个经法律授权的营养和健康声明，从英国和欧盟监管登记处翻译成21种语言；以及（ii）9088个由记者审核的公共健康声明，来自PUBHEALTH语料库，涵盖新冠疫情、堕胎、政治和一般健康，来自政府指导、科学摘要和媒体报道。模型通过多次运行的多数投票对每个声明进行支持或不支持的分类。准确性通过语言、主题、来源和模型进行分析。在授权声明上，准确性在英语和与其密切相关的主要欧洲语言中最高，并在几种广泛使用的非欧洲语言中下降，随着与英语的句法距离增加而下降。在现实世界中的公共健康声明上，准确性显著较低，并且系统性地根据主题和来源变化。模型在新冠疫情和政府归因的声明上表现最好，在一般健康和科学摘要上表现最差。在英语上的高性能掩盖了显著的上下文依赖性差距。训练数据暴露差异、编辑框架和主题特定的调优可能促成了这些差异，其幅度与跨语言差异相当。在健康声明验证中，LLM的准确性强烈依赖于语言、主题和信息来源。英语语言表现无法在不同上下文中可靠地推广，这强调了在部署到公共卫生沟通之前进行多语言、领域特定评估的必要性。
英文摘要：
Large language models (LLMs) are increasingly used to provide health advice, yet evidence on how their accuracy varies across languages, topics and information sources remains limited. We assess how linguistic and contextual factors affect the accuracy of AI-based health-claim verification. We evaluated seven widely used LLMs on two datasets: (i) 1,975 legally authorised nutrition and health claims from UK and EU regulatory registers translated into 21 languages; and (ii) 9,088 journalist-vetted public-health claims from the PUBHEALTH corpus spanning COVID-19, abortion, politics and general health, drawn from government advisories, scientific abstracts and media sources. Models classified each claim as supported or unsupported using majority voting across repeated runs. Accuracy was analysed by language, topic, source and model. Accuracy on authorised claims was highest in English and closely related European languages and declined in several widely spoken non-European languages, decreasing with syntactic distance from English. On real-world public-health claims, accuracy was substantially lower and varied systematically by topic and source. Models performed best on COVID-19 and government-attributed claims and worst on general health and scientific abstracts. High performance on English, canonical health claims masks substantial context-dependent gaps. Differences in training data exposure, editorial framing and topic-specific tuning likely contribute to these disparities, which are comparable in magnitude to cross-language differences. LLM accuracy in health-claim verification depends strongly on language, topic and information source. English-language performance does not reliably generalise across contexts, underscoring the need for multilingual, domain-specific evaluation before deployment in public-health communication.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
