爱丁堡临床NLP团队参加SemEval-2024任务2:除非能使用GPT-4,否则请微调你的模型
Edinburgh Clinical NLP at SemEval-2024 Task 2: Fine-tune your model unless you have access to GPT-4
AI总结:
本文研究临床试验NLI任务中LLM的推理一致性,提出合并三元组与语言建模目标微调的PEFT适配器方法,虽提升F1和一致性,但仍不及GPT-4。
摘要:
NLI4CT任务评估自然语言推理系统在预测假设是否蕴含或矛盾于临床试验报告证据方面的能力。在本研究中,我们采用多种策略评估了各种大型语言模型(LLMs),包括思维链、上下文学习和参数高效微调(PEFT)。我们提出了一种PEFT方法,通过合并分别使用三元组目标和语言建模目标微调的适配器来提高LLMs的一致性。我们发现,合并这两个PEFT适配器提高了LLMs的F1分数(+0.0346)和一致性(+0.152)。然而,在忠实性和一致性方面,我们的新方法并未产生比GPT-4更准确的结果。对三项指标取平均后,GPT-4以0.8328的成绩在竞赛中并列第一。最后,我们使用GPT-4进行的污染分析表明不存在测试数据泄漏。
英文摘要:
The NLI4CT task assesses Natural Language Inference systems in predicting whether hypotheses entail or contradict evidence from Clinical Trial Reports. In this study, we evaluate various Large Language Models (LLMs) with multiple strategies, including Chain-of-Thought, In-Context Learning, and Parameter-Efficient Fine-Tuning (PEFT). We propose a PEFT method to improve the consistency of LLMs by merging adapters that were fine-tuned separately using triplet and language modelling objectives. We found that merging the two PEFT adapters improves the F1 score (+0.0346) and consistency (+0.152) of the LLMs. However, our novel methods did not produce more accurate results than GPT-4 in terms of faithfulness and consistency. Averaging the three metrics, GPT-4 ranks joint-first in the competition with 0.8328. Finally, our contamination analysis with GPT-4 indicates that there was no test data leakage.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。