前沿大语言模型能否替代生物医学文本挖掘中的标注者？挑战分析与解决方案探索
Can Frontier LLMs Replace Annotators in Biomedical Text Mining? Analyzing Challenges and Exploring Solutions
AI总结：
该研究针对大语言模型在生物医学文本挖掘中表现欠佳的问题，分析三大挑战，开发动态提取标注指南指令的提示工程流水线，证明前沿LLM可媲美甚至超越SOTA BERT模型，还验证了LLM合成数据训练BERT及部分替代人工标注的可行性。
摘要：
既往多项研究表明，大语言模型（LLM）在生物医学文本挖掘中的表现欠佳。通过分析这些评估中的失败模式，我们明确了LLM在生物医学语料库中面临的三大核心挑战：（1）LLM无法从有监督数据中学习数据集特有的隐含细微差异；（2）判别性任务的通用格式要求限制了LLM的推理能力，尤其是缺乏测试时计算资源的LLM受影响更显著；（3）LLM难以遵循标注指南并匹配精确的标注模式，这阻碍了其理解详细标注要求的能力，而该能力在生物医学标注工作流中至关重要。我们针对上述问题试验了提示工程技术，并开发了一套可从标注指南中动态提取指令的流水线。实验结果显示，前沿LLM在仅需极少人工标注数据且无需微调的情况下，其性能可接近甚至超越基于BERT的当前最优（SOTA）模型。此外，我们对一个闭源LLM进行了模型蒸馏，证明仅用LLM标注的合成数据训练的BERT模型也能达到实用性能。基于这些发现，我们探索了在生物医学文本挖掘的生产场景中，用LLM部分替代人工标注的可行性。
英文摘要：
Multiple previous studies have reported suboptimal performance of LLMs in biomedical text mining. By analyzing failure patterns in these evaluations, we identified three primary challenges for LLMs in biomedical corpora: (1) LLMs fail to learn implicit dataset-specific nuances from supervised data, (2) The common formatting requirements of discriminative tasks limit the reasoning capabilities of LLMs particularly for LLMs that lack test-time compute, and (3) LLMs struggle to adhere to annotation guidelines and match exact schemas, which hinders their ability to understand detailed annotation requirements which is essential in biomedical annotation workflow. We experimented with prompt engineering techniques targeted to the above issues, and developed a pipeline that dynamically extracts instructions from annotation guidelines. Our results show that frontier LLMs can approach or surpass the performance of SOTA BERT-based models with minimal reliance on manually annotated data and without fine-tuning. Furthermore, we performed model distillation on a closed-source LLM, demonstrating that a BERT model trained exclusively on synthetic data annotated by LLMs can also achieve a practical performance. Based on these findings, we explored the feasibility of partially replacing manual annotation with LLMs in production scenarios for biomedical text mining.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
