Open Domain Question Answering (ODQA) within natural language processing involves building systems that answer factual questions using large-scale knowledge corpora. Recent advances stem from the confluence of several factors, such as large-scale training datasets, deep learning techniques, and the rise of large language models. High-quality datasets are used to train models on realistic scenarios and enable the evaluation of the system on potentially unseen data. Standardized metrics facilitate comparisons between different ODQA systems, allowing researchers to objectively track advancements in the field. Our study presents a thorough examination of the current landscape of ODQA benchmarking by reviewing 52 datasets and 20 evaluation techniques across textual and multimodal modalities. We introduce a novel taxonomy for ODQA datasets that incorporates both the modality and difficulty of the question types. Additionally, we present a structured organization of ODQA evaluation metrics along with a critical analysis of their inherent trade-offs. Our study aims to empower researchers by providing a framework for the robust evaluation of modern question-answering systems. We conclude by identifying the current challenges and outlining promising avenues for future research and development.
SRI International（斯坦福国际研究院）；Rutgers University（罗格斯大学）
Current pre-trained language models have lots of knowledge, but a more limited ability to use that knowledge. Bloom's Taxonomy helps educators teach children how to use knowledge by categorizing comprehension skills, so we use it to analyze and improve the comprehension skills of large pre-trained language models. Our experiments focus on zero-shot question answering, using the taxonomy to provide proximal context that helps the model answer questions by being relevant to those questions. We show targeting context in this manner improves performance across 4 popular common sense question answe…
University of Copenhagen（哥本哈根大学）；RIKEN（理化学研究所）；Microsoft Semantic Machines（微软语义机器公司）
Alongside huge volumes of research on deep learning models in NLP in the recent years, there has been also much work on benchmark datasets needed to track modeling progress. Question answering and reading comprehension have been particularly prolific in this regard, with over 80 new datasets appearing in the past two years. This study is the largest survey of the field to date. We provide an overview of the various formats and domains of the current resources, highlighting the current lacunae for future work. We further discuss the current classifications of "skills" that question answering/re…
University of Electronic Science and Technology of China（电子科技大学）
Visual question answering (VQA) is a challenging task, which has attracted more and more attention in the field of computer vision and natural language processing. However, the current visual question answering has the problem of language bias, which reduces the robustness of the model and has an adverse impact on the practical application of visual question answering. In this paper, we conduct a comprehensive review and analysis of this field for the first time, and classify the existing methods according to three categories, including enhancing visual information, weakening language priors, …
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Tencent AI Lab（腾讯AI实验室）
When answering a question, people often draw upon their rich world knowledge in addition to the particular context. While recent works retrieve supporting facts/evidence from commonsense knowledge bases to supply additional information to each question, there is still ample opportunity to advance it on the quality of the evidence. It is crucial since the quality of the evidence is the key to answering commonsense questions, and even determines the upper bound on the QA systems performance. In this paper, we propose a recursive erasure memory network (REM-Net) to cope with the quality improveme…
From Time Series Analysis to Question Answering: A Survey in the LLM Era
Renmin University of China（中国人民大学）；Tsinghua University（清华大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Aalborg University（奥胡斯大学）
Jadavpur University（贾达普大学）；Universitat Politècnica de València（瓦伦西亚理工大学）
Question classification (QC) is a prime constituent of automated question answering system. The work presented here demonstrates that the combination of multiple models achieve better classification performance than those obtained with existing individual models for the question classification task in Bengali. We have exploited state-of-the-art multiple model combination techniques, i.e., ensemble, stacking and voting, to increase QC accuracy. Lexical, syntactic and semantic features of Bengali questions are used for four well-known classifiers, namely Na\"ıve Bayes, kernel Na\"ıve Bayes, Rule…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；The Hong Kong University of Science and Technology（香港科技大学）
Taxonomy is not only a fundamental form of knowledge representation, but also crucial to vast knowledge-rich applications, such as question answering and web search. Most existing taxonomy construction methods extract hypernym-hyponym entity pairs to organize a "universal" taxonomy. However, these generic taxonomies cannot satisfy user's specific interest in certain areas and relations. Moreover, the nature of instance taxonomy treats each node as a single word, which has low semantic coverage. In this paper, we propose a method for seed-guided topical taxonomy construction, which takes a corp…
Textual Question Answering (QA) aims to provide precise answers to user's questions in natural language using unstructured data. One of the most popular approaches to this goal is machine reading comprehension(MRC). In recent years, many novel datasets and evaluation metrics based on classical MRC tasks have been proposed for broader textual QA tasks. In this paper, we survey 47 recent textual QA benchmark datasets and propose a new taxonomy from an application point of view. In addition, We summarize 8 evaluation metrics of textual QA tasks. Finally, we discuss current trends in constructing …
This paper introduces a new similarity measure based on edge counting in a taxonomy like WorldNet or Ontology. Measurement of similarity between text segments or concepts is very useful for many applications like information retrieval, ontology matching, text mining, and question answering and so on. Several measures have been developed for measuring similarity between two concepts: out of these we see that the measure given by Wu and Palmer [1] is simple, and gives good performance. Our measure is based on their measure but strengthens it. Wu and Palmer [1] measure has a disadvantage that it …
For companies with customer service, mapping intents inside their conversational data is crucial in building applications based on natural language understanding (NLU). Nevertheless, there is no established automated technique to gather the intents from noisy online chats or voice transcripts. Simple clustering approaches are not suited to intent-sparse dialogues. To solve this intent-landscape task, we propose an unsupervised pipeline that extracts the intents and the taxonomy of intents from real-world dialogues. Our pipeline mines intent-span candidates with an extractive Question-Answering…
Automatic construction of a taxonomy supports many applications in e-commerce, web search, and question answering. Existing taxonomy expansion or completion methods assume that new concepts have been accurately extracted and their embedding vectors learned from the text corpus. However, one critical and fundamental challenge in fixing the incompleteness of taxonomies is the incompleteness of the extracted concepts, especially for those whose names have multiple words and consequently low frequency in the corpus. To resolve the limitations of extraction-based methods, we propose GenTaxo to enha…
Information visualizations such as bar charts and line charts are very common for analyzing data and discovering critical insights. Often people analyze charts to answer questions that they have in mind. Answering such questions can be challenging as they often require a significant amount of perceptual and cognitive effort. Chart Question Answering (CQA) systems typically take a chart and a natural language question as input and automatically generate the answer to facilitate visual data analysis. Over the last few years, there has been a growing body of literature on the task of CQA. In this…
Alibaba Group（阿里巴巴集团）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
Question Answering (QA) over Knowledge Base (KB) aims to automatically answer natural language questions via well-structured relation information between entities stored in knowledge bases. In order to make KBQA more applicable in actual scenarios, researchers have shifted their attention from simple questions to complex questions, which require more KB triples and constraint inference. In this paper, we introduce the recent advances in complex QA. Besides traditional methods relying on templates and rules, the research is categorized into a taxonomy that contains two main branches, namely Inf…
Question Answering (QA) is one of the most important natural language processing (NLP) tasks. It aims using NLP technologies to generate a corresponding answer to a given question based on the massive unstructured corpus. With the development of deep learning, more and more challenging QA datasets are being proposed, and lots of new methods for solving them are also emerging. In this paper, we investigate influential QA datasets that have been released in the era of deep learning. Specifically, we begin with introducing two of the most common QA tasks - textual question answer and visual quest…
Multi-hop Question Answering under Temporal Knowledge Editing
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；South China University of Technology（华南理工大学）；Ant Group（蚂蚁集团）；SDAIA-KAUST AI（沙特数据与人工智能局-阿卜杜拉国王科技大学人工智能项目）；Provable Responsible AI and Data Analytics (PRADA) Lab（可信赖负责任人工智能与数据分析实验室）
The Ohio State University（俄亥俄州立大学）；Wright State University（莱特州立大学）；AFRL（美国空军研究实验室）
Visual Question Answering (VQA) systems are tasked with answering natural language questions corresponding to a presented image. Traditional VQA datasets typically contain questions related to the spatial information of objects, object attributes, or general scene questions. Recently, researchers have recognized the need to improve the balance of such datasets to reduce the system's dependency on memorized linguistic features and statistical biases, while aiming for enhanced visual understanding. However, it is unclear whether any latent patterns exist to quantify and explain these failures. A…
Survey of Large Multimodal Model Datasets, Application Categories and Taxonomy
University of Washington（华盛顿大学）；New York University（纽约大学）；Columbia University（哥伦比亚大学）；Liverpool John Moores University（利物浦约翰摩尔大学）；Chennai Mathematical Institute（金奈数学研究所）；Birla Institute of Technology（比拉理工学院）
In the community question answering (CQA) system, the answer selection task aims to identify the best answer for a specific question, and thus is playing a key role in enhancing the service quality through recommending appropriate answers for new questions. Recent advances in CQA answer selection focus on enhancing the performance by incorporating the community information, particularly the expertise (previous answers) and authority (position in the social network) of an answerer. However, existing approaches for incorporating such information are limited in (a) only considering either the exp…
College of Engineering, Pune（浦那工程学院）；C-DAC（印度先进计算发展中心）
In today's digital age in the dawning era of big data analytics it is not the information but the linking of information through entities and actions which defines the discourse. Any textual data either available on the Internet off off-line (like newspaper data, Wikipedia dump, etc) is basically connect information which cannot be treated isolated for its wholesome semantics. There is a need for an automated retrieval process with proper information extraction to structure the data for relevant and fast text analytics. The first big challenge is the conversion of unstructured textual data to …
When Large Language Models Meet Law: Dual-Lens Taxonomy, Technical Advances, and Ethical Governance
School of Law, China University of Political Science and Law（中国政法大学法学院）；Zhejiang University of Finance and Economics Dongfang College（浙江财经大学东洋学院）；School of Information Management for Law, China University of Political Science and Law（中国政法大学信息管理法学院）；Department of Artificial Intelligence, Chung-Ang University（成均馆大学人工智能系）；Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University（香港理工大学数据科学与人工智能系）
The Ohio State University（俄亥俄州立大学）；Nanjing University（南京大学）
Recent advances in deep learning have greatly propelled the research on semantic parsing. Improvement has since been made in many downstream tasks, including natural language interface to web APIs, text-to-SQL generation, among others. However, despite the close connection shared with these tasks, research on question answering over knowledge bases (KBQA) has comparatively been progressing slowly. We identify and attribute this to two unique challenges of KBQA, schema-level complexity and fact-level complexity. In this survey, we situate KBQA in the broader literature of semantic parsing and g…
Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Institute for Intelligent Computing, Alibaba Group（阿里巴巴集团智能计算研究院）
Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applications
Renmin University of China（中国人民大学）；The Chinese University of Hong Kong（香港中文大学）；Shenzhen University（深圳大学）；City University of Hong Kong（香港城市大学）；Peking University（北京大学）；Massachusetts General Hospital and Harvard Medical School（麻省总医院和哈佛医学院）
MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answering
National Library of Medicine, Division of Intramural Research（国家医学图书馆，院内研究部）；University of Illinois at Urbana-Champaign, Department of Computer Science（伊利诺伊大学厄巴纳-香槟分校计算机科学系）；University of Michigan Medical School（密歇根大学医学院）
Beyond MedQA: Towards Real-world Clinical Decision Making in the Era of LLMs
Department of Computer Science, Emory University（埃默里大学计算机科学系）；Children’s Healthcare of Atlanta（亚特兰大儿童医院）；Nell Hodgson Woodruff School of Nursing, Emory University（埃默里大学奈尔·霍德森·伍德鲁夫护理学院）
Agentic AI in Healthcare & Medicine: A Seven-Dimensional Taxonomy for Empirical Evaluation of LLM-based Agents
Department of Computer Science, New York University, CIMS, New York, USA（纽约大学计算机科学系，CIMS，纽约，美国）；Department of Computer Science, Cleveland State University, Cleveland, USA（克利夫兰州立大学计算机科学系，克利夫兰，美国）
Nankai University（南开大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Tsinghua University（清华大学）；Jiangsu Normal University（江苏师范大学）
Entity linking (EL) is the process of linking entity mentions appearing in web text with their corresponding entities in a knowledge base. EL plays an important role in the fields of knowledge engineering and data mining, underlying a variety of downstream applications such as knowledge base population, content analysis, relation extraction, and question answering. In recent years, deep learning (DL), which has achieved tremendous success in various domains, has also been leveraged in EL methods to surpass traditional machine learning based methods and yield the state-of-the-art performance. I…
University of Dayton（代顿大学）；RWTH Aachen（亚琛工业大学）；Zerotha Research（泽罗萨研究中心）；Cerence GmbH（塞伦斯有限责任公司）
In the era of Big Knowledge Graphs, Question Answering (QA) systems have reached a milestone in their performance and feasibility. However, their applicability, particularly in specific domains such as the biomedical domain, has not gained wide acceptance due to their "black box" nature, which hinders transparency, fairness, and accountability of QA systems. Therefore, users are unable to understand how and why particular questions have been answered, whereas some others fail. To address this challenge, in this paper, we develop an automatic approach for generating explanations during various …
Conditional Rank Allocation for Taxonomy-Aware Medical Language Model Adaptation
National University of Singapore（新加坡国立大学）；Shanghai General Hospital（上海市第一人民医院）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）；University of Pennsylvania（宾夕法尼亚大学）；Shandong University（山东大学）；Syracuse University（雪城大学）
From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
Islamic University of Technology（伊斯兰科技大学）；Advanced Machine Intelligence Research Lab（高级机器智能研究实验室）；American International University（美国国际大学）；Techno International New Town
