Developing methods of automated inference that are able to provide users with compelling human-readable justifications for why the answer to a question is correct is critical for domains such as science and medicine, where user trust and detecting costly errors are limiting factors to adoption. One of the central barriers to training question answering models on explainable inference tasks is the lack of gold explanations to serve as training data. In this paper we present a corpus of explanations for standardized science exams, a recent challenge task for question answering. We manually construct a corpus of detailed explanations for nearly all publicly available standardized elementary science question (approximately 1,680 3rd through 5th grade questions) and represent these as "explanation graphs" -- sets of lexically overlapping sentences that describe how to arrive at the correct answer to a question through a combination of domain and world knowledge. We also provide an explanation-centered tablestore, a collection of semi-structured tables that contain the knowledge to construct these elementary science explanations. Together, these two knowledge resources map out a substantial portion of the knowledge required for answering and explaining elementary science exams, and provide both structured and free-text training data for the explainable inference task.
University of Manchester（曼彻斯特大学）；Idiap Research Institute（Idiap研究所）
This paper presents a novel framework for reconstructing multi-hop explanations in science Question Answering (QA). While existing approaches for multi-hop reasoning build explanations considering each question in isolation, we propose a method to leverage explanatory patterns emerging in a corpus of scientific explanations. Specifically, the framework ranks a set of atomic facts by integrating lexical relevance with the notion of unification power, estimated analysing explanations for similar questions in the corpus. An extensive evaluation is performed on the Worldtree corpus, integrating k-…
Creating explanations for answers to science questions is a challenging task that requires multi-hop inference over a large set of fact sentences. This year, to refocus the Textgraphs Shared Task on the problem of gathering relevant statements (rather than solely finding a single 'correct path'), the WorldTree dataset was augmented with expert ratings of 'relevance' of statements to each overall explanation. Our system, which achieved second place on the Shared Task leaderboard, combines initial statement retrieval; language models trained to predict the relevance scores; and ensembling of a n…
University of Manchester（曼彻斯特大学）；Department of Computer Science（计算机科学系）
We propose a novel approach for answering and explaining multiple-choice science questions by reasoning on grounding and abstract inference chains. This paper frames question answering as an abductive reasoning problem, constructing plausible explanations for each choice and then selecting the candidate with the best explanation as the final answer. Our system, ExplanationLP, elicits explanations by constructing a weighted graph of relevant facts for each candidate answer and extracting the facts that satisfy certain structural and semantic constraints. To extract the explanations, we employ a…
TIB Leibniz Information Centre for Science and Technology（TIB莱布尼茨科学与技术信息中心）；University of Bonn（波恩大学）
In multiple-choice exams, students select one answer from among typically four choices and can explain why they made that particular choice. Students are good at understanding natural language questions and based on their domain knowledge can easily infer the question's answer by 'connecting the dots' across various pertinent facts. Considering automated reasoning for elementary science question answering, we address the novel task of generating explanations for answers from human-authored facts. For this, we examine the practically scalable framework of feature-rich support vector machines le…
Explainable question answering for science questions is a challenging task that requires multi-hop inference over a large set of fact sentences. To counter the limitations of methods that view each query-document pair in isolation, we propose the LSTM-Interleaved Transformer which incorporates cross-document interactions for improved multi-hop ranking. The LIT architecture can leverage prior ranking positions in the re-ranking setting. Our model is competitive on the current leaderboard for the TextGraphs 2020 shared task, achieving a test-set MAP of 0.5607, and would have gained third place h…
Investigating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queries
Yale University（耶鲁大学）；University of Pennsylvania（宾夕法尼亚大学）；Human Language Technology Center of Excellence（语言技术中心卓越中心）；Johns Hopkins University（约翰霍普金斯大学）
CompactRAG: Reducing LLM Calls and Token Overhead in Multi-Hop Question Answering
State Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；Erik Jonsson School of Engineering and Computer Science, University of Texas at Dallas（埃里克·乔纳森工程与计算机科学学院，德克萨斯大学达拉斯分校）；Isoftstone Information Technology (Group) Co.,Ltd.（伊软石信息技术（集团）有限公司）；College of Electronic and Information Engineering, Tongji University（电子信息工程学院，同济大学）；School of Electronic Information, Central South University（电子信息学院，中南大学）
Overview of the NLPCC 2025 Shared Task 4: Multi-modal, Multilingual, and Multi-hop Medical Instructional Video Question Answering Challenge
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究所，中国科学院）；School of Computer Science and Technology, Beijing Institute of Technology（计算机科学与技术学院，北京理工大学）；School of Engineering, Westlake University（工程学院，西湖大学）
We propose a method to teach an automated agent to learn how to search for multi-hop paths of relations between entities in an open domain. The method learns a policy for directing existing information retrieval and machine reading resources to focus on relevant regions of a corpus. The approach formulates the learning problem as a Markov decision process with a state representation that encodes the dynamics of the search process and a reward structure that minimizes the number of documents that must be processed while still finding multi-hop paths. We implement the method in an actor-critic r…
Corpus-Guided Dual-Path Propagation for Graph Retrieval-Augmented Generation
College of Software Engineering, Southeast University（东南大学软件工程学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of Computer Network and Information Integration (Southeast University) Ministry of Education（东南大学计算机网络与信息集成教育部重点实验室）
As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset…
We consider open-domain queston answering (QA) where answers are drawn from either a corpus, a knowledge base (KB), or a combination of both of these. We focus on a setting in which a corpus is supplemented with a large but incomplete KB, and on questions that require non-trivial (e.g., ``multi-hop'') reasoning. We describe PullNet, an integrated framework for (1) learning what to retrieve (from the KB and/or corpus) and (2) reasoning with this heterogeneous information to find the best answer. PullNet uses an {iterative} process to construct a question-specific subgraph that contains informat…
MultihopSpatial: Multi-hop Compositional Spatial Reasoning Benchmark for Vision-Language Model
Electronics and Telecommunications Research Institute, South Korea（韩国电信研究院）；Korea Advanced Institute of Science and Technology, South Korea（韩国科学技术院）；Sungkyunkwan University, South Korea（成均馆大学）；DeepAuto, South Korea（DeepAuto）
University of Maryland（马里兰大学）；Microsoft Research（微软研究院）
Open-domain question answering answers a question based on evidence retrieved from a large corpus. State-of-the-art neural approaches require intermediate evidence annotations for training. However, such intermediate annotations are expensive, and methods that rely on them cannot transfer to the more common setting, where only question-answer pairs are available. This paper investigates whether models can learn to find evidence from a large corpus, with only distant supervision from answer labels for model training, thereby generating no additional annotation cost. We introduce a novel approac…
We propose a novel Chain Guided Retriever-reader ({\tt CGR}) framework to model the reasoning chain for multi-hop Science Question Answering. Our framework is capable of performing explainable reasoning without the need of any corpus-specific annotations, such as the ground-truth reasoning chain, or human-annotated entity mentions. Specifically, we first generate reasoning chains from a semantic graph constructed by Abstract Meaning Representation of retrieved evidence facts. A \textit{Chain-aware loss}, concerning both local and global chain information, is also designed to enable the generat…
StepChain GraphRAG: Reasoning Over Knowledge Graphs for Multi-Hop Question Answering
University of Technology Sydney, Australia（澳大利亚技术大学）；Data61, CSIRO, Australia（数据61，CSIRO）；School of Engineering, Edith Cowan University, Australia（埃迪斯·科温大学工程学院）；University of New South Wales, Australia（新南威尔士大学）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Knowledge-graph-based reasoning has drawn a lot of attention due to its interpretability. However, previous methods suffer from the incompleteness of the knowledge graph, namely the interested link or entity that can be missing in the knowledge graph(explicit missing). Also, most previous models assume the distance between the target and source entity is short, which is not true on a real-world KG like Freebase(implicit missing). The sensitivity to the incompleteness of KG and the incapability to capture the long-distance link between entities have limited the performance of these models on la…
University of California, Santa Barbara（加州大学圣巴巴拉分校）；IBM Research（IBM研究院）
A key challenge of multi-hop question answering (QA) in the open-domain setting is to accurately retrieve the supporting passages from a large corpus. Existing work on open-domain QA typically relies on off-the-shelf information retrieval (IR) techniques to retrieve \textbf{answer passages}, i.e., the passages containing the groundtruth answers. However, IR-based approaches are insufficient for multi-hop questions, as the topic of the second or further hops is not explicitly covered by the question. To resolve this issue, we introduce a new sub-problem of open-domain multi-hop QA, which aims t…
Department of Computer Science and Technology, BNRist, Tsinghua University（清华大学计算机科学与技术系、BNRist）；Cloud BU, Huawei Technologies（华为技术有限公司云BU）；Nanyang Technological University（南洋理工大学）
Multi-hop Question Answering (QA) is a challenging task because it requires precise reasoning with entity relations at every step towards the answer. The relations can be represented in terms of labels in knowledge graph (e.g., \textit{spouse}) or text in text corpus (e.g., \textit{they have been married for 26 years}). Existing models usually infer the answer by predicting the sequential relation path or aggregating the hidden graph features. The former is hard to optimize, and the latter lacks interpretability. In this paper, we propose TransferNet, an effective and transparent model for mul…
Center for Artificial Intelligence Research (CAiRE)（人工智能研究中心（CAiRE））；The Hong Kong University of Science and Technology（香港科技大学）
Cross-domain named entity recognition (NER) models are able to cope with the scarcity issue of NER samples in target domains. However, most of the existing NER benchmarks lack domain-specialized entity types or do not focus on a certain domain, leading to a less effective cross-domain evaluation. To address these obstacles, we introduce a cross-domain NER dataset (CrossNER), a fully-labeled collection of NER data spanning over five diverse domains with specialized entity categories for different domains. Additionally, we also provide a domain-related corpus since using it to continue pre-train…
Single and Multi-Hop Question-Answering Datasets for Reticular Chemistry with GPT-4-Turbo
University of California, Berkeley（加州大学伯克利分校）；Kavli Energy Nanoscience Institute（卡夫利能源纳米科学研究所）；Bakar Institute of Digital Materials for the Planet（巴卡尔地球数字材料研究所）；College of Computing, Data Science, and Society（计算、数据科学与社会学院）；University of California, Berkeley Library（加州大学伯克利分校图书馆）；King Abdulaziz City for Science and Technology（阿卜杜勒-阿齐兹国王科技城）；KACST−UC Berkeley Center of Excellence for Nanomaterials for Clean Energy Applications（KACST-加州大学伯克利分校清洁能源应用纳米材料卓越中心）
Allen Institute for AI（艾伦人工智能研究所）；University of Arizona（亚利桑那大学）
Composing knowledge from multiple pieces of texts is a key challenge in multi-hop question answering. We present a multi-hop reasoning dataset, Question Answering via Sentence Composition(QASC), that requires retrieving facts from a large corpus and composing them to answer a multiple-choice question. QASC is the first dataset to offer two desirable properties: (a) the facts to be composed are annotated in a large corpus, and (b) the decomposition into these facts is not evident from the question itself. The latter makes retrieval challenging as the system must introduce new concepts or relati…
Tsinghua University（清华大学）；University of Maryland（马里兰大学）
For humans and computers, the first step in answering an open-domain question is retrieving a set of relevant documents from a large corpus. However, the strategies that computers use fundamentally differ from those of humans. To better understand these differences, we design a gamified interface for data collection -- Cheater's Bowl -- where a human answers complex questions with access to both traditional and modern search tools. We collect a dataset of human search sessions, analyze human search strategies, and compare them to state-of-the-art multi-hop QA models. Humans query logically, ap…
Synthesize-on-Graph: Knowledgeable Synthetic Data Generation for Continue Pre-training of Large Language Models
DataArc Tech Ltd.（DataArc科技有限公司）；IDEA Research, International Digital Economy Academy（IDEA研究所、国际数字经济学院）；Gaoling School of Artificial Intelligence, Renmin University of China（法律人工智能学院，中国人民大学）
OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text
Harbin Institute of Technology（哈尔滨工业大学）；Nanjing University（南京大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；Tsinghua University（清华大学）
A Navigational Approach for Comprehensive RAG via Traversal over Proposition Graphs
Idiap Research Institute（日内瓦研究所）；École Polytechnique Fédérale de Lausanne（联邦理工学院洛桑分校）；University of Manchester（曼彻斯特大学）；CRUK Manchester Institute（CRUK曼彻斯特研究所）
M$^3$-Med: A Benchmark for Multi-lingual, Multi-modal, and Multi-hop Reasoning in Medical Instructional Video Understanding
Beijing Institute of Technology（北京理工大学）；The Hong Kong Polytechnic University（香港理工大学）；Chinese Academy of Sciences（中国科学院）；Air Force Hospital of Southern Theater Command of PLA（中国人民解放军南部战区空军医院）；Shenzhen Traditional Chinese Medicine Hospital（深圳中医药医院）
ABBYY；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；National Research University Higher School of Economics（俄罗斯国立高等经济大学）；Novosibirsk State University（新西伯利亚国立大学）；Lomonosov Moscow State University（莫斯科罗蒙诺索夫国立大学）；Innopolis University（伊诺波利斯大学）；Kazan Federal University（喀山联邦大学）
We show-case an application of information extraction methods, such as named entity recognition (NER) and relation extraction (RE) to a novel corpus, consisting of documents, issued by a state agency. The main challenges of this corpus are: 1) the annotation scheme differs greatly from the one used for the general domain corpora, and 2) the documents are written in a language other than English. Unlike expectations, the state-of-the-art transformer-based models show modest performance for both tasks, either when approached sequentially, or in an end-to-end fashion. Our experiments have demonst…
University of Chinese Academy of Sciences（中国科学院大学）；Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）
Information seeking is an essential step for open-domain question answering to efficiently gather evidence from a large corpus. Recently, iterative approaches have been proven to be effective for complex questions, by recursively retrieving new evidence at each step. However, almost all existing iterative approaches use predefined strategies, either applying the same retrieval function multiple times or fixing the order of different retrieval functions, which cannot fulfill the diverse requirements of various questions. In this paper, we propose a novel adaptive information-seeking strategy fo…
