评估开源大语言模型在自动化事实核查中的应用
Evaluating open-source Large Language Models for automated fact-checking
AI总结：
本研究通过三项实验评估开源大语言模型在自动化事实核查中的表现，发现其在识别声明与文章关联及核查已核查内容方面表现良好，但确认事实性新闻时不及RoBERTa等微调模型，且外部知识帮助有限，表明LLMs尚需改进才能替代人工核查。
摘要：
网络错误信息日益泛滥，推动了对自动化事实核查解决方案的需求。大语言模型（LLMs）已成为辅助这一任务的潜在工具，但其有效性仍不确定。本研究评估了多种开源LLMs的事实核查能力，重点关注它们在不同上下文信息水平下判断声明的能力。我们开展了三项关键实验：（1）评估LLMs能否识别声明与事实核查文章之间的语义关系；（2）评估模型在给定相关事实核查文章时验证声明的准确性；（3）测试LLMs在利用Google和Wikipedia等外部知识来源数据时的事实核查能力。结果表明，LLMs在识别声明与文章的关联以及核查已被事实核查过的报道方面表现良好，但在确认事实性新闻方面存在困难，其表现不及RoBERTa等传统微调模型。此外，引入外部知识并未显著提升LLMs的性能，这要求采用更具针对性的方法。我们的发现凸显了LLMs在自动化事实核查中的潜力与局限，强调在它们能够可靠替代人工事实核查员之前仍需进一步改进。
英文摘要：
The increasing prevalence of online misinformation has heightened the demand for automated fact-checking solutions. Large Language Models (LLMs) have emerged as potential tools for assisting in this task, but their effectiveness remains uncertain. This study evaluates the fact-checking capabilities of various open-source LLMs, focusing on their ability to assess claims with different levels of contextual information. We conduct three key experiments: (1) evaluating whether LLMs can identify the semantic relationship between a claim and a fact-checking article, (2) assessing models' accuracy in verifying claims when given a related fact-checking article, and (3) testing LLMs' fact-checking abilities when leveraging data from external knowledge sources such as Google and Wikipedia. Our results indicate that LLMs perform well in identifying claim-article connections and verifying fact-checked stories but struggle with confirming factual news, where they are outperformed by traditional fine-tuned models such as RoBERTa. Additionally, the introduction of external knowledge does not significantly enhance LLMs' performance, calling for more tailored approaches. Our findings highlight both the potential and limitations of LLMs in automated fact-checking, emphasizing the need for further refinements before they can reliably replace human fact-checkers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
