FactFinders at CheckThat! 2024: 通过数据剪枝利用大型语言模型优化值得核查的陈述检测
FactFinders at CheckThat! 2024: Refining Check-worthy Statement Detection with LLMs through Data Pruning
AI总结：
本研究利用八种开源大语言模型结合微调和提示工程，通过两步数据剪枝方法，在CheckThat! 2024英语任务中高效识别值得核查的陈述，仅用44%训练数据即获第一名。
摘要：
社交媒体和互联网信息的快速传播给事实核查带来了重大挑战，尤其是在识别值得核查的声明方面，即从大量句子中筛选出需要事实核查的声明。这一挑战凸显了确定声明优先级的需求，即哪些声明值得进行事实核查。尽管近年来该领域取得了进展，但大型语言模型（LLMs）如GPT的应用直到最近才引起研究关注。然而，许多开源LLMs仍未得到充分探索。因此，本研究探讨了八种主流开源LLMs在微调和提示工程下的应用，以从政治转录文本中识别值得核查的陈述。此外，我们提出了一种两步数据剪枝方法，自动识别高质量的训练数据实例，以实现有效学习。我们方法的效率通过在CheckThat! 2024的值得核查性估计任务中英语数据集的评估得到证明。此外，使用数据剪枝进行的实验表明，仅使用约44%的训练数据即可实现有竞争力的性能。我们的团队在英语值得核查性估计任务中排名第一。
英文摘要：
The rapid dissemination of information through social media and the Internet has posed a significant challenge for fact-checking, among others in identifying check-worthy claims that fact-checkers should pay attention to, i.e. filtering claims needing fact-checking from a large pool of sentences. This challenge has stressed the need to focus on determining the priority of claims, specifically which claims are worth to be fact-checked. Despite advancements in this area in recent years, the application of large language models (LLMs), such as GPT, has only recently drawn attention in studies. However, many open-source LLMs remain underexplored. Therefore, this study investigates the application of eight prominent open-source LLMs with fine-tuning and prompt engineering to identify check-worthy statements from political transcriptions. Further, we propose a two-step data pruning approach to automatically identify high-quality training data instances for effective learning. The efficiency of our approach is demonstrated through evaluations on the English language dataset as part of the check-worthiness estimation task of CheckThat! 2024. Further, the experiments conducted with data pruning demonstrate that competitive performance can be achieved with only about 44\% of the training data. Our team ranked first in the check-worthiness estimation task in the English language.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
