As the first step of automatic fact checking, claim check-worthiness detection is a critical component of fact checking systems. There are multiple lines of research which study this problem: check-worthiness ranking from political speeches and debates, rumour detection on Twitter, and citation needed detection from Wikipedia. To date, there has been no structured comparison of these various tasks to understand their relatedness, and no investigation into whether or not a unified approach to all of them is achievable. In this work, we illuminate a central challenge in claim check-worthiness detection underlying all of these tasks, being that they hinge upon detecting both how factual a sentence is, as well as how likely a sentence is to be believed without verification. As such, annotators only mark those instances they judge to be clear-cut check-worthy. Our best performing method is a unified approach which automatically corrects for this using a variant of positive unlabelled learning that finds instances which were incorrectly labelled as not check-worthy. In applying this, we out-perform the state of the art in two of the three tasks studied for claim check-worthiness detection in English.
TOBB University of Economics and Technology（TOBB经济与技术大学）
The massive amount of misinformation spreading on the Internet on a daily basis has enormous negative impacts on societies. Therefore, we need automated systems helping fact-checkers in the combat against misinformation. In this paper, we propose a model prioritizing the claims based on their check-worthiness. We use BERT model with additional features including domain-specific controversial topics, word embeddings, and others. In our experiments, we show that our proposed model outperforms all state-of-the-art models in both test collections of CLEF Check That! Lab in 2018 and 2019. We also c…
SiteGround；Amazon（亚马逊）；Qatar Computing Research Institute, HBKU（哈马德·本·哈利法大学卡塔尔计算研究所）；Qatar University（卡塔尔大学）；College of Humanities and Social Sciences, HBKU（哈马德·本·哈利法大学人文与社会科学学院）；Sofia University “St Kliment Ohridski”（索非亚大学“圣克里门特·奥赫里德斯基”）
We present an overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims, with focus on Task 1: Check-Worthiness. The task asks to predict which claims in a political debate should be prioritized for fact-checking. In particular, given a debate or a political speech, the goal was to produce a ranked list of its sentences based on their worthiness for fact checking. We offered the task in both English and Arabic, based on debates from the 2016 US Presidential Campaign, as well as on some speeches during and after the campaign. A total of 30 teams r…
Daffodil International University（达福迪尔国际大学）；Qatar Computing Research Institute（卡塔尔计算研究所）
The wide use of social media and digital technologies facilitates sharing various news and information about events and activities. Despite sharing positive information misleading and false information is also spreading on social media. There have been efforts in identifying such misleading information both manually by human experts and automatic tools. Manual effort does not scale well due to the high volume of information, containing factual claims, are appearing online. Therefore, automatically identifying check-worthy claims can be very useful for human experts. In this study, we describe …
Automatic fact-checking systems detect misinformation, such as fake news, by (i) selecting check-worthy sentences for fact-checking, (ii) gathering related information to the sentences, and (iii) inferring the factuality of the sentences. Most prior research on (i) uses hand-crafted features to select check-worthy sentences, and does not explicitly account for the recent finding that the top weighted terms in both check-worthy and non-check-worthy sentences are actually overlapping [15]. Motivated by this, we present a neural check-worthiness sentence ranking model that represents each word in…
Fraunhofer SIT at CheckThat! 2023: Tackling Classification Uncertainty Using Model Souping on the Example of Check-Worthiness Classification
Fraunhofer Institute for Secure Information Technology SIT（弗劳恩霍夫安全信息技术研究所）；ATHENE - National Research Center for Applied Cybersecurity（ATHENE——国家应用网络安全研究中心）
Fraunhofer SIT 团队在 CLEF-2023 CheckThat! Task 1B 中提出基于 Model Souping 的集成分类方法，用于政治辩论文本的值得核查性分类，在英语数据集上取得 0.878 的 F1 分数，排名第二。
Università di Bologna（博洛尼亚大学）；Qatar University（卡塔尔大学）；Qatar Computing Research Institute, HBKU（卡塔尔计算研究所（哈马德·本·哈利法大学））；Sofia University “St Kliment Ohridski”（索菲亚大学“圣克莱门特·奥赫里德斯基”）
We present an overview of the third edition of the CheckThat! Lab at CLEF 2020. The lab featured five tasks in two different languages: English and Arabic. The first four tasks compose the full pipeline of claim verification in social media: Task 1 on check-worthiness estimation, Task 2 on retrieving previously fact-checked claims, Task 3 on evidence retrieval, and Task 4 on claim verification. The lab is completed with Task 5 on check-worthiness estimation in political debates and speeches. A total of 67 teams registered to participate in the lab (up from 47 at CLEF 2019), and 23 of them actu…
Sofia University（索非亚大学）；University of Copenhagen（哥本哈根大学）；Amazon Core ML（亚马逊核心机器学习部门）；Università di Bologna（博洛尼亚大学）；Qatar Computing Research Institute, HBKU（哈马德·本·哈利法大学卡塔尔计算研究所）
We propose a multi-task deep-learning approach for estimating the check-worthiness of claims in political debates. Given a political debate, such as the 2016 US Presidential and Vice-Presidential ones, the task is to predict which statements in the debate should be prioritized for fact-checking. While different fact-checking organizations would naturally make different choices when analyzing the same debate, we show that it pays to learn from multiple sources simultaneously (PolitiFact, FactCheck, ABC, CNN, NPR, NYT, Chicago Tribune, The Guardian, and Washington Post) in a multi-task learning …
PRHLT Research Center, Universitat Politècnica de València（瓦伦西亚理工大学PRHLT研究中心）；DW Innovation（德国之声创新部）；University of Marburg（马尔堡大学）
Check-worthiness detection is the task of identifying claims, worthy to be investigated by fact-checkers. Resource scarcity for non-world languages and model learning costs remain major challenges for the creation of models supporting multilingual check-worthiness detection. This paper proposes cross-training adapters on a subset of world languages, combined by adapter fusion, to detect claims emerging globally in multiple languages. (1) With a vast number of annotators available for world languages and the storage-efficient adapter models, this approach is more cost efficient. Models can be u…
TIB Leibniz Information Centre for Science and Technology（莱布尼茨科技信息中心）；L3S Research Center, Leibniz University Hannover（汉诺威莱布尼茨大学L3S研究中心）
In this digital age of news consumption, a news reader has the ability to react, express and share opinions with others in a highly interactive and fast manner. As a consequence, fake news has made its way into our daily life because of very limited capacity to verify news on the Internet by large companies as well as individuals. In this paper, we focus on solving two problems which are part of the fact-checking ecosystem that can help to automate fact-checking of claims in an ever increasing stream of content on social media. For the first problem, claim check-worthiness prediction, we explo…
Identifying check-worthy claims is often the first step of automated fact-checking systems. Tackling this task in a multilingual setting has been understudied. Encoding inputs with multilingual text representations could be one approach to solve the multilingual check-worthiness detection. However, this approach could suffer if cultural bias exists within the communities on determining what is check-worthy.In this paper, we propose a language identification task as an auxiliary task to mitigate unintended bias.With this purpose, we experiment joint training by using the datasets from CLEF-2021…
Misinformation spread over social media has become an undeniable infodemic. However, not all spreading claims are made equal. If propagated, some claims can be destructive, not only on the individual level, but to organizations and even countries. Detecting claims that should be prioritized for fact-checking is considered the first step to fight against spread of fake news. With training data limited to a handful of languages, developing supervised models to tackle the problem over lower-resource languages is currently infeasible. Therefore, our work aims to investigate whether we can use exis…
Queen Mary University of London（伦敦玛丽女王大学）；Taif University（塔伊夫大学）
An important component of an automated fact-checking system is the claim check-worthiness detection system, which ranks sentences by prioritising them based on their need to be checked. Despite a body of research tackling the task, previous research has overlooked the challenging nature of identifying check-worthy claims across different topics. In this paper, we assess and quantify the challenge of detecting check-worthy claims for new, unseen topics. After highlighting the problem, we propose the AraCWA model to mitigate the performance deterioration when detecting check-worthy claims across…
TIB – Leibniz Information Centre for Science and Technology（莱布尼茨科技与信息研究所）；L3S Research Center, Leibniz University Hannover（汉诺威莱布尼茨大学L3S研究中心）
Fake news is a severe problem in social media. In this paper, we present an empirical study on visual, textual, and multimodal models for the tasks of claim, claim check-worthiness, and conspiracy detection, all of which are related to fake news detection. Recent work suggests that images are more influential than text and often appear alongside fake text. To this end, several multimodal models have been proposed in recent years that use images along with text to detect fake news on social media sites like Twitter. However, the role of images is not well understood for claim detection, specifi…
From Chaos to Clarity: Claim Normalization to Empower Fact-Checking
Indraprastha Institute of Information Technology Delhi（因德拉普拉斯塔信息技术学院德里校区）；Mohammed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Indian Institute of Technology Delhi（印度理工学院德里校区）
While recent work on automated fact-checking has focused mainly on verifying and explaining claims, for which the list of claims is readily available, identifying check-worthy claim sentences from a text remains challenging. Current claim identification models rely on manual annotations for each sentence in the text, which is an expensive task and challenging to conduct on a frequent basis across multiple domains. This paper explores methodology to identify check-worthy claim sentences from fake news articles, irrespective of domain, without explicit sentence-level annotations. We leverage two…
The CLEF-2025 CheckThat! Lab: Subjectivity, Fact-Checking, Claim Normalization, and Retrieval
Qatar Computing Research Institute（卡塔尔计算研究所）；University of Applied Sciences Potsdam（波茨坦应用科学大学）；University of Bologna（博洛尼亚大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Indian Institute of Technology Delhi（印度理工学院德里分校）；GESIS - Leibniz Institute for the Social Sciences（GESIS莱布尼茨社会科学研究所）；Heinrich-Heine-University Düsseldorf（杜塞尔多夫海因里希·海涅大学）；Delft University of Technology（代尔夫特理工大学）；Indraprastha Institute of Information Technology（因德拉普拉斯塔信息技术学院）；University of Montpellier（蒙彼利埃大学）
Computer Science and Engineering Department, Qatar University（卡塔尔大学计算机科学与工程系）；Qatar Computing Research Institute, HBKU（哈马德·本·哈利法大学卡塔尔计算研究所）；DIT, Università di Bologna（博洛尼亚大学DIT（此处DIT按原文保留，若需全称需更多上下文））；Department of Computer Science, University of Copenhagen（哥本哈根大学计算机科学系）
We present an overview of the second edition of the CheckThat! Lab at CLEF 2019. The lab featured two tasks in two different languages: English and Arabic. Task 1 (English) challenged the participating systems to predict which claims in a political debate or speech should be prioritized for fact-checking. Task 2 (Arabic) asked to (A) rank a given set of Web pages with respect to a check-worthy claim based on their usefulness for fact-checking that claim, (B) classify these same Web pages according to their degree of usefulness for fact-checking the target claim, (C) identify useful passages fr…
We study the problem of automatic fact-checking, paying special attention to the impact of contextual and discourse information. We address two related tasks: (i) detecting check-worthy claims, and (ii) fact-checking claims. We develop supervised systems based on neural networks, kernel-based support vector machines, and combinations thereof, which make use of rich input representations in terms of discourse cues and contextual features. For the check-worthiness estimation task, we focus on political debates, and we model the target claim in the context of the full intervention of a participan…
The past decade has seen a substantial rise in the amount of mis- and disinformation online, from targeted disinformation campaigns to influence politics, to the unintentional spreading of misinformation about public health. This development has spurred research in the area of automatic fact checking, from approaches to detect check-worthy claims and determining the stance of tweets towards claims, to methods to determine the veracity of claims given evidence documents. These automatic methods are often content-based, using natural language processing methods, which in turn utilise deep neural…
The CLEF-2026 CheckThat! Lab: Advancing Multilingual Fact-Checking
University of Applied Sciences Potsdam, Germany（应用科学大学波茨坦，德国）；GESIS - Leibniz Institute for the Social Sciences, Cologne, Germany（GESIS—社会科学院莱比锡研究所，德国）；Stockholm University, Sweden（斯德哥尔摩大学，瑞典）；University of Stavanger, Norway（斯塔万格大学，挪威）；Indian Institute of Technology Delhi, India（印度理工学院德里，印度）；Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi（穆罕默德·本·扎耶德人工智能大学，阿布扎比）；Delft University of Technology, The Netherlands（代尔夫特理工大学，荷兰）；Independent Researcher（独立研究者）；médialab Sciences Po, Paris, France（Sciences Po 媒介实验室，法国）；University of Montpellier, LIRMM, CNRS, Montpellier, France（蒙彼利埃大学，LIRMM，CNRS，法国）
