Recent works integrating Knowledge Graphs (KGs) have shown promising improvements in enhancing the reasoning capabilities of Large Language Models (LLMs). However, existing benchmarks primarily focus on closed-ended tasks, leaving a gap in evaluating performance on more complex, real-world scenarios. This limitation also hinders a thorough assessment of KGs' potential to reduce hallucinations in LLMs. To address this, we introduce OKGQA, a new benchmark specifically designed to evaluate LLMs augmented with KGs in open-ended, real-world question answering settings. OKGQA reflects practical complexities through diverse question types and incorporates metrics to quantify both hallucination rates and reasoning improvements in LLM+KG models. To consider the scenarios in which KGs may contain varying levels of errors, we propose a benchmark variant, OKGQA-P, to assess model performance when the semantics and structure of KGs are deliberately perturbed and contaminated. In this paper, we aims to (1) explore whether KGs can make LLMs more trustworthy in an open-ended setting, and (2) conduct a comparative analysis to shed light on method design. We believe this study can facilitate a more complete performance comparison and encourages continuous improvement in integrating KGs with LLMs to mitigate hallucination, and make LLMs more trustworthy. Code and data are released at https://github.com/Y-Sui/OKGQA.
University of Rochester（罗切斯特大学）；Adobe Research（Adobe研究院）
Answering open-ended questions is an essential capability for any intelligent agent. One of the most interesting recent open-ended question answering challenges is Visual Question Answering (VQA) which attempts to evaluate a system's visual understanding through its answers to natural language questions about images. There exist many approaches to VQA, the majority of which do not exhibit deeper semantic understanding of the candidate answers they produce. We study the importance of generating plausible answers to a given question by introducing the novel task of `Answer Proposal': for a given…
REC-CBM: Rubric-Aware Error-Correction Concept Bottleneck Models for Trustworthy Open-Ended Grading
School of Computing and Augmented Intelligence, Arizona State University, USA（计算与增强智能学院，亚利桑那州立大学，美国）；Mary Lou Fulton Teachers College, Arizona State University, USA（玛丽·卢·福洛顿教师学院，亚利桑那州立大学，美国）；Department of Computer Science, National Yang Ming Chiao Tung University, TW（国立阳明交通大学计算机科学系，台湾）
Video question answering has recently received a lot of attention from multimodal video researchers. Most video question answering datasets are usually in the form of multiple-choice. But, the model for the multiple-choice task does not infer the answer. Rather it compares the answer candidates for picking the correct answer. Furthermore, it makes it difficult to extend to other tasks. In this paper, we challenge the existing multiple-choice video question answering by changing it to open-ended video question answering. To tackle open-ended question answering, we use the pretrained GPT2 model.…
O$^2$-Searcher: A Searching-based Agent Model for Open-Domain Open-Ended Question Answering
Zhejiang University（浙江大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Fudan University（复旦大学）；State Key Laboratory of Industrial Control Technology（工业控制技术国家重点实验室）
The Hong Kong University of Science and Technology（香港科技大学）
Considerable advancements have been made in various NLP tasks based on the impressive power of large language models (LLMs) and many NLP applications are deployed in our daily lives. In this work, we challenge the capability of LLMs with the new task of Ethical Quandary Generative Question Answering. Ethical quandary questions are more challenging to address because multiple conflicting answers may exist to a single quandary. We explore the current capability of LLMs in providing an answer with a deliberative exchange of different perspectives to an ethical quandary, in the approach of Socrati…
ORCA: Open-ended Response Correctness Assessment for Audio Question Answering
Brno University of Technology（布尔诺科技大学）；Universidad Autónoma de Madrid（马德里自治大学）；University of Buenos Aires（布宜诺斯艾利斯大学）；Tufts University（塔夫茨大学）；University of Maryland（马里兰大学）
Chain-of-Discussion: A Multi-Model Framework for Complex Evidence-Based Question Answering
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Center for Data Science, Peking University（北京大学数据科学中心）；Key Laboratory of Intelligent Press Media Technology, Peking University（北京大学智能新闻媒体技术重点实验室）
SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluation
GreenDynamics Pty. Ltd（格林动力私人有限公司）；City University of Hong Kong（香港城市大学）；University of Chicago（芝加哥大学）；University of Sydney（悉尼大学）；University of New South Wales（新南威尔士大学）
RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering
Raidium；Université de Paris Cité, Hôpital Européen Georges Pompidou, AP-HP（巴黎西岱大学，乔治·蓬皮杜欧洲医院，AP-HP）；Department of Vascular and Oncological Interventional Radiology, INSERM（血管与肿瘤介入放射学系，法国国家健康与医学研究院）
Hybrid Retrieval-Augmented Generation Agent for Trustworthy Legal Question Answering in Judicial Forensics
Department of Electronic Engineering, City University of Hong Kong(DongGuan)（香港城市大学（东莞）电子工程系）；Department of Electronic Engineering, City University of Hong Kong（香港城市大学电子工程系）
Path-RAG: Knowledge-Guided Key Region Retrieval for Open-ended Pathology Visual Question Answering
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Arizona State University（亚利桑那州立大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Massachusetts Institute of Technology（麻省理工学院）
Large Language Models in the Clinic: A Comprehensive Benchmark
University of Oxford（牛津大学）；Amazon（亚马逊公司）；Harvard T.H. Chan School of Public Health（哈佛陈曾熙公共卫生学院）；Institut polytechnique de Paris（巴黎综合理工学院）；Oxford-Suzhou Centre for Advanced Research（牛津-苏州高等研究院）
In this work, we propose a deep neural architecture that uses an attention mechanism which utilizes region based image features, the natural language question asked, and semantic knowledge extracted from the regions of an image to produce open-ended answers for questions asked in a visual question answering (VQA) task. The combination of both region based features and region based textual information about the image bolsters a model to more accurately respond to questions and potentially do so with less required training data. We evaluate our proposed architecture on a VQA task against a stron…
Video Question Answering is a challenging problem in visual information retrieval, which provides the answer to the referenced video content according to the question. However, the existing visual question answering approaches mainly tackle the problem of static image question, which may be ineffectively for video question answering due to the insufficiency of modeling the temporal dynamics of video contents. In this paper, we study the problem of video question answering by modeling its temporal dynamics with frame-level attention mechanism. We propose the attribute-augmented attention networ…
MediQAl: A French Medical Question Answering Dataset for Knowledge and Reasoning Evaluation
Data Clinic, University Hospital of Nantes, France（南特大学医院数据诊所，法国）；Nantes Université, École Centrale Nantes, CNRS, LS2N, France（南特大学，中央理工学院南特分校，国家科学研究中心，LS2N，法国）
DermaBench: A Clinician-Annotated Benchmark Dataset for Dermatology Visual Question Answering and Reasoning
Imperial College London（帝国理工学院伦敦分校）；Istanbul Medeniyet University（伊斯坦布尔医学大学）；Usak Research and Training Hospital（Usak研究与培训医院）；Istanbul Research and Training Hospital（伊斯坦布尔研究与培训医院）；Ipswich Hospital（伊普斯韦奇医院）；Medicana Atakoy Hospital（Medicana阿塔基医院）
We introduce NExT-QA, a rigorously designed video question answering (VideoQA) benchmark to advance video understanding from describing to explaining the temporal actions. Based on the dataset, we set up multi-choice and open-ended QA tasks targeting causal action reasoning, temporal action reasoning, and common scene comprehension. Through extensive analysis of baselines and established VideoQA techniques, we find that top-performing methods excel at shallow scene descriptions but are weak in causal and temporal action reasoning. Furthermore, the models that are effective on multi-choice QA, …
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Progress in language and image understanding by machines has sparkled the interest of the research community in more open-ended, holistic tasks, and refueled an old AI dream of building intelligent machines. We discuss a few prominent challenges that characterize such holistic tasks and argue for "question answering about images" as a particular appealing instance of such a holistic task. In particular, we point out that it is a version of a Turing Test that is likely to be more robust to over-interpretations and contrast it with tasks like grounding and generation of descriptions. Finally, we…
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Washington（华盛顿大学）；Stanford University（斯坦福大学）；UMass Amherst（马萨诸塞大学阿默斯特分校）
We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total). The dialogs involve two crowd workers: (1) a student who poses a sequence of freeform questions to learn as much as possible about a hidden Wikipedia text, and (2) a teacher who answers the questions by providing short excerpts from the text. QuAC introduces challenges not found in existing machine comprehension datasets: its questions are often more open-ended, unanswerable, or only meaningful within the dialog context, as we show in a detailed qualitative e…
