arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.20514cs.CLcs.AI
收藏

在资源有限环境中评估大型语言模型对历史健康危机知识的评估:一种混合多指标研究

Evaluating Large Language Models on Historical Health Crisis Knowledge in Resource-Limited Settings: A Hybrid Multi-Metric Study

organization= Department of Electrical \& Computer Engineering , addressline= North South University , city= Dhaka , country= Bangladesh

Mohammed Rakibul Hasan

AI总结:

本研究评估GPT-4、Gemini Pro、Llama 3和Mistral-7B在低资源环境下对新冠、登革热、尼帕病毒和登革热等健康危机问题的处理能力,通过语义相似度、专家评估和NLI方法分析模型表现,揭示LLM在流行病学历史和健康危机知识表示中的优缺点。

摘要:

大型语言模型(LLMs)在提供健康信息方面具有显著潜力。然而,其在低资源环境中的可靠性仍存疑。本研究评估GPT-4、Gemini Pro、Llama~3和Mistral-7B在低资源环境下(如孟加拉国)对新冠、登革热、尼帕病毒和登革热等健康危机问题的处理能力。我们从权威来源构建了问题-答案数据集,并通过语义相似度、专家-模型交叉评估和自然语言推理(NLI)评估模型输出。研究结果突显了LLM在代表流行病学历史和健康危机知识方面的优势与局限性,强调了其在资源有限环境中指导政策制定的潜力与风险。

英文摘要:

Large Language Models (LLMs) offer significant potential for delivering health information. However, their reliability in low-resource contexts remains uncertain. This study evaluates GPT-4, Gemini Pro, Llama~3, and Mistral-7B on health crisis-related enquiries concerning COVID-19, dengue, the Nipah virus, and Chikungunya in the low-resource context of Bangladesh. We constructed a question--answer dataset from authoritative sources and assessed model outputs through semantic similarity, expert-model cross-evaluation, and Natural Language Inference (NLI). Findings highlight both the strengths and limitations of LLMs in representing epidemiological history and health crisis knowledge, underscoring their promise and risks for informing policy in resource-constrained environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑