arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.09429cs.CL
收藏

开放(临床)大语言模型对指令措辞具有敏感性

Open (Clinical) LLMs are Sensitive to Instruction Phrasings

Northeastern University(东北大学);Codametrix(科达美创)

Alberto Mario Ceballos Arroyo, Monica Munnangi, Jiuding Sun, Karen Y. C. Zhang, Denis Jered McInerney, Byron C. Wallace, Silvio Amir

AI总结:

本研究针对临床NLP场景下指令微调LLMs对自然指令措辞变化的鲁棒性问题,通过收集医生提供的多任务提示测试7款模型,发现通用与临床专用模型均存在性能波动,且专用模型更脆弱,措辞差异还会影响公平性。

摘要:

经过指令微调的大语言模型(LLMs)在接收到自然语言指令时可执行广泛任务,但它们对这类指令的措辞方式较为敏感。这一问题在医疗健康领域尤其值得关注,因为临床医生不太可能是经验丰富的提示工程师,且该领域中输出不准确的潜在后果更为严重。\n 这引出了一个实际问题:经过指令微调的LLMs对临床自然语言处理(NLP)任务指令的自然变化鲁棒性如何?我们收集了来自医生的、覆盖多种任务的提示,量化了7款LLMs(部分为通用模型,其余为专用模型)对自然(即非对抗性)指令措辞的敏感性。我们发现所有模型的性能都存在显著差异,且或许令人意外的是,与通用领域的同类模型相比,专门基于临床数据训练的领域特定模型尤为脆弱。此外,任意的措辞差异可能影响公平性,例如,用于死亡率预测的有效但表述不同的指令,会在整体性能以及不同人口统计群体间的差异方面产生波动范围。

英文摘要:

Instruction-tuned Large Language Models (LLMs) can perform a wide range of tasks given natural language instructions to do so, but they are sensitive to how such instructions are phrased. This issue is especially concerning in healthcare, as clinicians are unlikely to be experienced prompt engineers and the potential consequences of inaccurate outputs are heightened in this domain. This raises a practical question: How robust are instruction-tuned LLMs to natural variations in the instructions provided for clinical NLP tasks? We collect prompts from medical doctors across a range of tasks and quantify the sensitivity of seven LLMs -- some general, others specialized -- to natural (i.e., non-adversarial) instruction phrasings. We find that performance varies substantially across all models, and that -- perhaps surprisingly -- domain-specific models explicitly trained on clinical data are especially brittle, compared to their general domain counterparts. Further, arbitrary phrasing differences can affect fairness, e.g., valid but distinct instructions for mortality prediction yield a range both in overall performance, and in terms of differences between demographic groups.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑