arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.00484cs.CL
收藏

爱丁堡临床NLP团队参加SemEval-2024任务2:除非能使用GPT-4,否则请微调你的模型

Edinburgh Clinical NLP at SemEval-2024 Task 2: Fine-tune your model unless you have access to GPT-4

School of Informatics, University of Edinburgh(爱丁堡大学信息学院);Usher Institute, University of Edinburgh(爱丁堡大学厄舍研究所);Edinburgh Futures Institute, University of Edinburgh(爱丁堡未来学院);School of Literatures, Languages and Cultures, University of Edinburgh(爱丁堡大学文学、语言与文化学院)

Aryo Pradipta Gema, Giwon Hong, Pasquale Minervini, Luke Daines, Beatrice Alex

AI总结:

本文研究临床试验NLI任务中LLM的推理一致性,提出合并三元组与语言建模目标微调的PEFT适配器方法,虽提升F1和一致性,但仍不及GPT-4。

摘要:

NLI4CT任务评估自然语言推理系统在预测假设是否蕴含或矛盾于临床试验报告证据方面的能力。在本研究中,我们采用多种策略评估了各种大型语言模型(LLMs),包括思维链、上下文学习和参数高效微调(PEFT)。我们提出了一种PEFT方法,通过合并分别使用三元组目标和语言建模目标微调的适配器来提高LLMs的一致性。我们发现,合并这两个PEFT适配器提高了LLMs的F1分数(+0.0346)和一致性(+0.152)。然而,在忠实性和一致性方面,我们的新方法并未产生比GPT-4更准确的结果。对三项指标取平均后,GPT-4以0.8328的成绩在竞赛中并列第一。最后,我们使用GPT-4进行的污染分析表明不存在测试数据泄漏。

英文摘要:

The NLI4CT task assesses Natural Language Inference systems in predicting whether hypotheses entail or contradict evidence from Clinical Trial Reports. In this study, we evaluate various Large Language Models (LLMs) with multiple strategies, including Chain-of-Thought, In-Context Learning, and Parameter-Efficient Fine-Tuning (PEFT). We propose a PEFT method to improve the consistency of LLMs by merging adapters that were fine-tuned separately using triplet and language modelling objectives. We found that merging the two PEFT adapters improves the F1 score (+0.0346) and consistency (+0.152) of the LLMs. However, our novel methods did not produce more accurate results than GPT-4 in terms of faithfulness and consistency. Averaging the three metrics, GPT-4 ranks joint-first in the competition with 0.8328. Finally, our contamination analysis with GPT-4 indicates that there was no test data leakage.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑