Fact verification (FV) is a challenging task which aims to verify a claim using multiple evidential sentences from trustworthy corpora, e.g., Wikipedia. Most existing approaches follow a three-step pipeline framework, including document retrieval, sentence retrieval and claim verification. High-quality evidences provided by the first two steps are the foundation of the effective reasoning in the last step. Despite being important, high-quality evidences are rarely studied by existing works for FV, which often adopt the off-the-shelf models to retrieve relevant documents and sentences in an "index-retrieve-then-rank" fashion. This classical approach has clear drawbacks as follows: i) a large document index as well as a complicated search process is required, leading to considerable memory and computational overhead; ii) independent scoring paradigms fail to capture the interactions among documents and sentences in ranking; iii) a fixed number of sentences are selected to form the final evidence set. In this work, we propose GERE, the first system that retrieves evidences in a generative fashion, i.e., generating the document titles as well as evidence sentence identifiers. This enables us to mitigate the aforementioned technical issues since: i) the memory and computational cost is greatly reduced because the document index is eliminated and the heavy ranking process is replaced by a light generative process; ii) the dependency between documents and that between sentences could be captured via sequential generation process; iii) the generative formulation allows us to dynamically select a precise set of relevant evidences for each claim. The experimental results on the FEVER dataset show that GERE achieves significant improvements over the state-of-the-art baselines, with both time-efficiency and memory-efficiency.
Document retrieval is a core component of many knowledge-intensive natural language processing task formulations such as fact verification and question answering. Sources of textual knowledge, such as Wikipedia articles, condition the generation of answers from the models. Recent advances in retrieval use sequence-to-sequence models to incrementally predict the title of the appropriate Wikipedia page given a query. However, this method requires supervision in the form of human annotation to label which Wikipedia pages contain appropriate context. This paper introduces a distant-supervision met…
A key component of fact verification is thevevidence retrieval, often from multiple documents. Recent approaches use dense representations and condition the retrieval of each document on the previously retrieved ones. The latter step is performed over all the documents in the collection, requiring storing their dense representations in an index, thus incurring a high memory footprint. An alternative paradigm is retrieve-and-rerank, where documents are retrieved using methods such as BM25, their sentences are reranked, and further documents are retrieved conditioned on these sentences, reducing…
School of Information, The University of Arizona（亚利桑那大学信息学院）；Amazon Alexa AI（亚马逊 Alexa AI）
Recent work has shown that an answer verification step introduced in Transformer-based answer selection models can significantly improve the state of the art in Question Answering. This step is performed by aggregating the embeddings of top $k$ answer candidates to support the verification of a target answer. Although the approach is intuitive and sound still shows two limitations: (i) the supporting candidates are ranked only according to the relevancy with the question and not with the answer, and (ii) the support provided by the other answer candidates is suboptimal as these are retrieved i…
Hangzhou Dianzi University（杭州电子科技大学）；University of Yamanashi（山梨大学）
Scientific claim verification can help the researchers to easily find the target scientific papers with the sentence evidence from a large corpus for the given claim. Some existing works propose pipeline models on the three tasks of abstract retrieval, rationale selection and stance prediction. Such works have the problems of error propagation among the modules in the pipeline and lack of sharing valuable information among modules. We thus propose an approach, named as ARSJoint, that jointly learns the modules for the three tasks with a machine reading comprehension framework by including clai…
Authorship attribution, being an important problem in many areas in-cluding information retrieval, computational linguistics, law and journalism etc., has been identified as a subject of increasingly research interest in the re-cent years. In case of Author Identification task in PAN at CLEF 2015, the main focus was given on cross-genre and cross-topic author verification tasks. We have used several word-based and style-based features to identify the dif-ferences between the known and unknown problems of one given set and label the unknown ones accordingly using a Random Forest based classifie…
SERC: LDPC-Inspired Semantic Error Correction for Retrieval-Augmented Generation
Department of Information Communications Engineering, Hankuk University of Foreign Studies, Republic of Korea（韩国外国语大学信息通信工程系）；Division of Computer Engineering, Hankuk University of Foreign Studies, Republic of Korea（韩国外国语大学计算机工程系）；Department of Statistics, Hankuk University of Foreign Studies, Republic of Korea（韩国外国语大学统计学系）
Information Sciences Institute（信息科学研究所）；University of Southern California（南加利福尼亚大学）；University of Nebraska-Lincoln（内布拉斯加大学林肯分校）
Automatic fact verification has become an increasingly popular topic in recent years and among datasets the Fact Extraction and VERification (FEVER) dataset is one of the most popular. In this work we present BEVERS, a tuned baseline system for the FEVER dataset. Our pipeline uses standard approaches for document retrieval, sentence selection, and final claim classification, however, we spend considerable effort ensuring optimal performance for each component. The results are that BEVERS achieves the highest FEVER score and label accuracy among all systems, published or unpublished. We also ap…
Complex deep learning models now achieve state of the art performance for many document retrieval tasks. The best models process the query or claim jointly with the document. However for fast scalable search it is desirable to have document embeddings which are independent of the claim. In this paper we show that knowledge distillation can be used to encourage a model that generates claim independent document encodings to mimic the behavior of a more complex model which generates claim dependent encodings. We explore this approach in document retrieval for a fact extraction and verification ta…
CLAIR-Fin: An Adversarial Multi-Agent Framework for Claim-Level Verification and Adaptive Debate in Cross-Modal Financial QA
Ahsanullah University of Science and Technology（阿萨努拉科技大学）；Jashore University of Science and Technology（杰索尔科技大学）；American International University - Bangladesh（孟加拉国美国国际大学）
School of Information, The University of Arizona（亚利桑那大学信息学院）；Amazon Alexa AI（亚马逊Alexa人工智能部门）
This paper studies joint models for selecting correct answer sentences among the top $k$ provided by answer sentence selection (AS2) modules, which are core components of retrieval-based Question Answering (QA) systems. Our work shows that a critical step to effectively exploit an answer set regards modeling the interrelated information between pair of answers. For this purpose, we build a three-way multi-classifier, which decides if an answer supports, refutes, or is neutral with respect to another one. More specifically, our neural architecture integrates a state-of-the-art AS2 model with th…
StateRAG: Typed State Contracts for Complex Retrieval-Augmented Generation
School of Computer Science and Engineering, Northeastern University, China（东北大学计算机科学与工程学院）；Tsinghua University, China（清华大学）；Kunming University of Science and Technology, China（昆明理工大学）；NiuTrans Research, Shenyang, China（沈阳NiuTrans研究院）
This article investigates multilingual evidence retrieval and fact verification as a step to combat global disinformation, a first effort of this kind, to the best of our knowledge. The goal is building multilingual systems that retrieve in evidence-rich languages to verify claims in evidence-poor languages that are more commonly targeted by disinformation. To this end, our EnmBERT fact verification system shows evidence of transfer learning ability and 400 example mixed English-Romanian dataset is made available for cross-lingual transfer learning evaluation.
MEVER: Multi-Modal and Explainable Claim Verification with Graph-based Evidence Retrieval
University of Sheffield（谢菲尔德大学）；University of Science and Technology Beijing（北京科技大学）；University of California San Diego（加州大学圣地亚哥分校）；The Pennsylvania State University（宾夕法尼亚州立大学）
Informatics Institute（信息学研究所）；University of Amsterdam（阿姆斯特丹大学）
Motivated by the promising performance of pre-trained language models, we investigate BERT in an evidence retrieval and claim verification pipeline for the FEVER fact extraction and verification challenge. To this end, we propose to use two BERT models, one for retrieving potential evidence sentences supporting or rejecting claims, and another for verifying claims based on the predicted evidence sets. To train the BERT retrieval system, we use pointwise and pairwise loss functions, and examine the effect of hard negative mining. A second BERT model is trained to classify the samples as support…
Research Center for Social Computing and Information Retrieval（社会计算与信息检索研究中心）；Harbin Institute of Technology（哈尔滨工业大学）
Table-based fact verification task aims to verify whether the given statement is supported by the given semi-structured table. Symbolic reasoning with logical operations plays a crucial role in this task. Existing methods leverage programs that contain rich logical information to enhance the verification process. However, due to the lack of fully supervised signals in the program generation process, spurious programs can be derived and employed, which leads to the inability of the model to catch helpful logical operations. To address the aforementioned problems, in this work, we formulate the …
Verifying the veracity of claims requires reasoning over a large knowledge base, often in the form of corpora of trustworthy sources. A common approach consists in retrieving short portions of relevant text from the reference documents and giving them as input to a natural language inference module that determines whether the claim can be inferred or contradicted from them. This approach, however, struggles when multiple pieces of evidence need to be collected and combined from different documents, since the single documents are often barely related to the target claim and hence they are left …
An Evidence-Grounded Retrieval-Augmented Transformer Framework for Health Misinformation Verification
Yobe State University（约贝州立大学）；Yobe AI Research Laboratory (YAIR Lab)（约贝人工智能研究实验室）；Novosibirsk State University（新西伯利亚国立大学）；College of Computer Science and Engineering, Hamad Bin Khalifa University（哈马德·本·哈利法大学计算机科学与工程学院）
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
The increasing concern with misinformation has stimulated research efforts on automatic fact checking. The recently-released FEVER dataset introduced a benchmark fact-verification task in which a system is asked to verify a claim using evidential sentences from Wikipedia documents. In this paper, we present a connected system consisting of three homogeneous neural semantic matching models that conduct document retrieval, sentence selection, and claim verification jointly for fact extraction and verification. For evidence retrieval (document retrieval and sentence selection), unlike traditional…
The Fact Extraction and VERification (FEVER) shared task was launched to support the development of systems able to verify claims by extracting supporting or refuting facts from raw text. The shared task organizers provide a large-scale dataset for the consecutive steps involved in claim verification, in particular, document retrieval, fact extraction, and claim classification. In this paper, we present our claim verification pipeline approach, which, according to the preliminary results, scored third in the shared task, out of 23 competing systems. For the document retrieval, we implemented a…
Veri-R1: Toward Precise and Faithful Claim Verification via Online Reinforcement Learning
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Fudan University（复旦大学）；Tsinghua University（清华大学）；Hong Kong University of Science and Technology（香港科学与技术大学）
University of Mannheim（曼海姆大学）；Thapar Institute of Engineering & Technology（塔帕尔工程技术学院）；Charles University（查理大学）
Exponential growth in digital information outlets and the race to publish has made scientific misinformation more prevalent than ever. However, the task to fact-verify a given scientific claim is not straightforward even for researchers. Scientific claim verification requires in-depth knowledge and great labor from domain experts to substantiate supporting and refuting evidence from credible scientific sources. The SciFact dataset and corresponding task provide a benchmarking leaderboard to the community to develop automatic scientific claim verification systems via extracting and assimilating…
RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Wuhan University of Science and Technology（武汉科技大学）；National University of Singapore（新加坡国立大学）；Northeastern University at Qinhuangdao（东北大学秦皇岛分校）；BOSS Zhipin（BOSS直聘）
