arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.15944cs.CL
收藏

AutoMedPrompt:一种利用文本梯度优化LLM医学提示的新框架

AutoMedPrompt: A New Framework for Optimizing LLM Medical Prompts Using Textual Gradients

Keck Data Science Institute, Pepperdine University(佩珀代因大学凯克数据科学研究所);Department of Medicine, University of California, Los Angeles(加利福尼亚大学洛杉矶分校医学院);Biological Sciences Division, University of Chicago(芝加哥大学生物科学部);Brain Research Institute, University of California, Los Angeles(加利福尼亚大学洛杉矶分校脑研究所)

Sean Wu, Michael Koo, Fabien Scalzo, Ira Kurtz

AI总结:

AutoMedPrompt利用文本梯度优化系统提示,提升开源LLM在医学问答基准上的性能,创下PubMedQA新SOTA(82.6%),并超越专有模型。

摘要:

大型语言模型(LLMs)在医学及其他知识领域展现出日益复杂的性能。传统创建专家级LLM的方法需要对大型数据集进行广泛的微调和模型训练。近期,提示工程(而非微调)已显示出提升通用基础模型性能的潜力。然而,诸如思维链(CoT)之类的提示方法可能不适用于所有亚专科,且k-shot方法可能向上下文空间引入不相关标记。我们提出了AutoMedPrompt,探索利用文本梯度通过系统提示优化来引发医学相关推理。AutoMedPrompt利用TextGrad的基于文本的自动微分来提升通用基础LLM的能力。我们在开源LLM Llama 3上,使用多个问答基准(包括MedQA、PubMedQA和肾脏病学亚专科专用NephSAP)评估了AutoMedPrompt。我们的结果表明,使用文本梯度的提示在开源LLM上优于先前方法,并超越了专有模型如GPT-4、Claude 3 Opus和Med-PaLM 2。AutoMedPrompt在PubMedQA上以82.6%的准确率创下新的最先进(SOTA)性能,同时在MedQA(77.7%)和NephSAP(63.8%)上亦优于先前针对开源模型的提示策略。

英文摘要:

Large language models (LLMs) have demonstrated increasingly sophisticated performance in medical and other fields of knowledge. Traditional methods of creating specialist LLMs require extensive fine-tuning and training of models on large datasets. Recently, prompt engineering, instead of fine-tuning, has shown potential to boost the performance of general foundation models. However, prompting methods such as chain-of-thought (CoT) may not be suitable for all subspecialty, and k-shot approaches may introduce irrelevant tokens into the context space. We present AutoMedPrompt, which explores the use of textual gradients to elicit medically relevant reasoning through system prompt optimization. AutoMedPrompt leverages TextGrad's automatic differentiation via text to improve the ability of general foundation LLMs. We evaluated AutoMedPrompt on Llama 3, an open-source LLM, using several QA benchmarks, including MedQA, PubMedQA, and the nephrology subspecialty-specific NephSAP. Our results show that prompting with textual gradients outperforms previous methods on open-source LLMs and surpasses proprietary models such as GPT-4, Claude 3 Opus, and Med-PaLM 2. AutoMedPrompt sets a new state-of-the-art (SOTA) performance on PubMedQA with an accuracy of 82.6$\%$, while also outperforming previous prompting strategies on open-sourced models for MedQA (77.7$\%$) and NephSAP (63.8$\%$).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑