大语言模型投票：面向罕见病识别的提示方法
Large Language Models Vote: Prompting for Rare Disease Identification
AI总结：
本文提出 Models-Vote Prompting（MVP）方法，通过提示多个大语言模型执行同一任务并进行多数投票，在单样本罕见病识别与分类中优于任一单一模型，并发布新数据集及自动化评估方案。
摘要：
生成式大语言模型（LLMs）的出现凸显了对准确且高效提示方法的需求。LLMs 常被应用于少样本学习（FSL）场景中，即在极少量训练数据下执行任务。FSL 已在许多人工智能（AI）子领域中流行，包括面向健康的 AI。罕见病仅影响人群中很小一部分。由于数据可用性有限，从临床记录中识别罕见病本质上需要 FSL 技术。人工数据收集和标注既昂贵又耗时。在本文中，我们提出了 Models-Vote Prompting（MVP），这是一种灵活的提示方法，用于在 FSL 设置下提升 LLM 查询的性能。MVP 的工作原理是提示多个 LLM 执行相同任务，然后对所得输出进行多数投票。该方法在单样本罕见病识别和分类任务上取得了优于集成中任何单一模型的结果。我们还发布了一个面向 FSL 的新型罕见病数据集，可供签署 MIMIC-IV 数据使用协议（DUA）的人员使用。此外，在使用 MVP 时，每个模型会被多次提示，这大幅增加了人工标注所需的时间，为解决这一问题，我们评估了使用 JSON 自动化生成式 LLM 评估的可行性。
英文摘要：
The emergence of generative Large Language Models (LLMs) emphasizes the need for accurate and efficient prompting approaches. LLMs are often applied in Few-Shot Learning (FSL) contexts, where tasks are executed with minimal training data. FSL has become popular in many Artificial Intelligence (AI) subdomains, including AI for health. Rare diseases affect a small fraction of the population. Rare disease identification from clinical notes inherently requires FSL techniques due to limited data availability. Manual data collection and annotation is both expensive and time-consuming. In this paper, we propose Models-Vote Prompting (MVP), a flexible prompting approach for improving the performance of LLM queries in FSL settings. MVP works by prompting numerous LLMs to perform the same tasks and then conducting a majority vote on the resulting outputs. This method achieves improved results to any one model in the ensemble on one-shot rare disease identification and classification tasks. We also release a novel rare disease dataset for FSL, available to those who signed the MIMIC-IV Data Use Agreement (DUA). Furthermore, in using MVP, each model is prompted multiple times, substantially increasing the time needed for manual annotation, and to address this, we assess the feasibility of using JSON for automating generative LLM evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
