使用大语言模型根据CIViC证据模型标注医学论文
Using LLMs to label medical papers according to the CIViC evidence model
AI总结：
本文提出医学NLP多标签分类任务CIViC Evidence，通过微调BERT、RoBERTa及领域预训练模型并与GPT-4少样本对比，证明领域预训练Transformer效果最佳且优于GPT-4和逻辑回归。
摘要：
我们将序列分类问题CIViC Evidence引入医学自然语言处理（NLP）领域。CIViC Evidence指的是一个多标签分类问题，其任务是为科学论文摘要分配临床证据标签，这些论文研究了基因组变异、癌症类型和治疗方法的多种组合。我们使用不同的语言模型来处理CIViC Evidence：我们在CIViC Evidence数据集上微调BERT和RoBERTa的预训练检查点，并用在领域特定文本上预训练的相同架构模型来挑战它们的性能。在此背景下，我们发现BiomedBERT和BioLinkBERT在CIViC Evidence上可以超越BERT（类支持加权F1分数的绝对提升分别为+0.8%和+0.9%）。与在二元组tf-idf分数上训练的逻辑回归相比，所有基于Transformer的模型都表现出明显的性能优势（F1分数提高1.5%至2.7%）。我们将上述类似BERT的模型与OpenAI的GPT-4在少样本设置下（在我们原始测试数据集的一个小子集上）进行比较，结果表明，在没有额外提示工程或微调的情况下，GPT-4在CIViC Evidence上的表现比我们的六个微调模型更差（加权F1分数为66.1%，而最佳微调模型为71.8%）。然而，其性能合理接近在二元组tf-idf分数上训练的逻辑回归模型的基准（加权F1分数为67.7%）。
英文摘要：
We introduce the sequence classification problem CIViC Evidence to the field of medical NLP. CIViC Evidence denotes the multi-label classification problem of assigning labels of clinical evidence to abstracts of scientific papers which have examined various combinations of genomic variants, cancer types, and treatment approaches. We approach CIViC Evidence using different language models: We fine-tune pretrained checkpoints of BERT and RoBERTa on the CIViC Evidence dataset and challenge their performance with models of the same architecture which have been pretrained on domain-specific text. In this context, we find that BiomedBERT and BioLinkBERT can outperform BERT on CIViC Evidence (+0.8% and +0.9% absolute improvement in class-support weighted F1 score). All transformer-based models show a clear performance edge when compared to a logistic regression trained on bigram tf-idf scores (+1.5 - 2.7% improved F1 score). We compare the aforementioned BERT-like models to OpenAI's GPT-4 in a few-shot setting (on a small subset of our original test dataset), demonstrating that, without additional prompt-engineering or fine-tuning, GPT-4 performs worse on CIViC Evidence than our six fine-tuned models (66.1% weighted F1 score compared to 71.8% for the best fine-tuned model). However, performance gets reasonably close to the benchmark of a logistic regression model trained on bigram tf-idf scores (67.7% weighted F1 score).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
