In this paper, we introduce a robotic agent specifically designed to analyze external environments and address participants' questions. The primary focus of this agent is to assist individuals using language-based interactions within video-based scenes. Our proposed method integrates video recognition technology and natural language processing models within the robotic agent. We investigate the crucial factors affecting human-robot interactions by examining pertinent issues arising between participants and robot agents. Methodologically, our experimental findings reveal a positive relationship between trust and interaction efficiency. Furthermore, our model demonstrates a 2\% to 3\% performance enhancement in comparison to other benchmark methods.
Video question answering has recently received a lot of attention from multimodal video researchers. Most video question answering datasets are usually in the form of multiple-choice. But, the model for the multiple-choice task does not infer the answer. Rather it compares the answer candidates for picking the correct answer. Furthermore, it makes it difficult to extend to other tasks. In this paper, we challenge the existing multiple-choice video question answering by changing it to open-ended video question answering. To tackle open-ended question answering, we use the pretrained GPT2 model.…
University of Rochester（罗切斯特大学）；Adobe Research（Adobe研究院）
Answering open-ended questions is an essential capability for any intelligent agent. One of the most interesting recent open-ended question answering challenges is Visual Question Answering (VQA) which attempts to evaluate a system's visual understanding through its answers to natural language questions about images. There exist many approaches to VQA, the majority of which do not exhibit deeper semantic understanding of the candidate answers they produce. We study the importance of generating plausible answers to a given question by introducing the novel task of `Answer Proposal': for a given…
The multi-relational Knowledge Base Question Answering (KBQA) system performs multi-hop reasoning over the knowledge graph (KG) to achieve the answer. Recent approaches attempt to introduce the knowledge graph embedding (KGE) technique to handle the KG incompleteness but only consider the triple facts and neglect the significant semantic correlation between paths and multi-relational questions. In this paper, we propose a Path and Knowledge Embedding-Enhanced multi-relational Question Answering model (PKEEQA), which leverages multi-hop paths between entities in the KG to evaluate the ambipolar…
The Hong Kong University of Science and Technology（香港科技大学）
Considerable advancements have been made in various NLP tasks based on the impressive power of large language models (LLMs) and many NLP applications are deployed in our daily lives. In this work, we challenge the capability of LLMs with the new task of Ethical Quandary Generative Question Answering. Ethical quandary questions are more challenging to address because multiple conflicting answers may exist to a single quandary. We explore the current capability of LLMs in providing an answer with a deliberative exchange of different perspectives to an ethical quandary, in the approach of Socrati…
O$^2$-Searcher: A Searching-based Agent Model for Open-Domain Open-Ended Question Answering
Zhejiang University（浙江大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Fudan University（复旦大学）；State Key Laboratory of Industrial Control Technology（工业控制技术国家重点实验室）
ORCA: Open-ended Response Correctness Assessment for Audio Question Answering
Brno University of Technology（布尔诺科技大学）；Universidad Autónoma de Madrid（马德里自治大学）；University of Buenos Aires（布宜诺斯艾利斯大学）；Tufts University（塔夫茨大学）；University of Maryland（马里兰大学）
Various data mining tasks have been proposed to study Community Question Answering (CQA) platforms like Stack Overflow. The relatedness between some of these tasks provides useful learning signals to each other via Multi-Task Learning (MTL). However, due to the high heterogeneity of these tasks, few existing works manage to jointly solve them in a unified framework. To tackle this challenge, we develop a multi-relational graph based MTL model called Heterogeneous Multi-Task Graph Isomorphism Network (HMTGIN) which efficiently solves heterogeneous CQA tasks. In each training forward pass, HMTGI…
Question Answering (QA), as a research field, has primarily focused on either knowledge bases (KBs) or free text as a source of knowledge. These two sources have historically shaped the kinds of questions that are asked over these sources, and the methods developed to answer them. In this work, we look towards a practical use-case of QA over user-instructed knowledge that uniquely combines elements of both structured QA over knowledge bases, and unstructured QA over narrative, introducing the task of multi-relational QA over personal narrative. As a first step towards this goal, we make three …
Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Heuritech（赫瑞泰克公司）；Conservatoire National des Arts et Métiers（法国国立工艺学院）
Multimodal attentional networks are currently state-of-the-art models for Visual Question Answering (VQA) tasks involving real images. Although attention allows to focus on the visual content relevant to the question, this simple mechanism is arguably insufficient to model complex reasoning features required for VQA or other high-level tasks. In this paper, we propose MuRel, a multimodal relational network which is learned end-to-end to reason over real images. Our first contribution is the introduction of the MuRel cell, an atomic reasoning primitive representing interactions between question…
Technical University of Denmark（丹麦技术大学）；Tradeshift；DeepMind
This paper is concerned with learning to solve tasks that require a chain of interdependent steps of relational inference, like answering complex questions about the relationships between objects, or solving puzzles where the smaller elements of a solution mutually constrain each other. We introduce the recurrent relational network, a general purpose module that operates on a graph representation of objects. As a generalization of Santoro et al. [2017]'s relational network, it can augment any neural network model with the capacity to do many-step relational reasoning. We achieve state of the a…
Semantic parsing, as an important approach to question answering over knowledge bases (KBQA), transforms a question into the complete query graph for further generating the correct logical query. Existing semantic parsing approaches mainly focus on relations matching with paying less attention to the underlying internal structure of questions (e.g., the dependencies and relations between all entities in a question) to select the query graph. In this paper, we present a relational graph convolutional network (RGCN)-based model gRGCN for semantic parsing in KBQA. gRGCN extracts the global semant…
Path-RAG: Knowledge-Guided Key Region Retrieval for Open-ended Pathology Visual Question Answering
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Arizona State University（亚利桑那州立大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Massachusetts Institute of Technology（麻省理工学院）
RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering
Raidium；Université de Paris Cité, Hôpital Européen Georges Pompidou, AP-HP（巴黎西岱大学，乔治·蓬皮杜欧洲医院，AP-HP）；Department of Vascular and Oncological Interventional Radiology, INSERM（血管与肿瘤介入放射学系，法国国家健康与医学研究院）
Chain-of-Discussion: A Multi-Model Framework for Complex Evidence-Based Question Answering
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Center for Data Science, Peking University（北京大学数据科学中心）；Key Laboratory of Intelligent Press Media Technology, Peking University（北京大学智能新闻媒体技术重点实验室）
SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluation
GreenDynamics Pty. Ltd（格林动力私人有限公司）；City University of Hong Kong（香港城市大学）；University of Chicago（芝加哥大学）；University of Sydney（悉尼大学）；University of New South Wales（新南威尔士大学）
We introduce NExT-QA, a rigorously designed video question answering (VideoQA) benchmark to advance video understanding from describing to explaining the temporal actions. Based on the dataset, we set up multi-choice and open-ended QA tasks targeting causal action reasoning, temporal action reasoning, and common scene comprehension. Through extensive analysis of baselines and established VideoQA techniques, we find that top-performing methods excel at shallow scene descriptions but are weak in causal and temporal action reasoning. Furthermore, the models that are effective on multi-choice QA, …
Centre for AI Research（人工智能研究中心）；University of Agder（阿格德尔大学）；University of Pittsburgh（匹兹堡大学）
TMs are a pattern recognition approach that uses finite state machines for learning and propositional logic to represent patterns. In addition to being natively interpretable, they have provided competitive accuracy for various tasks. In this paper, we increase the computing power of TMs by proposing a first-order logic-based framework with Herbrand semantics. The resulting TM is relational and can take advantage of logical structures appearing in natural language, to learn rules that represent how actions and consequences are related in the real world. The outcome is a logic program of Horn c…
Topology of Reasoning: Retrieved Cell Complex-Augmented Generation for Textual Graph Question Answering
Academy of Advanced Interdisciplinary Studies, Chongqing University of Posts and Telecommunications（重庆邮电大学先进交叉学科研究院）；School of Computer Science and Technology, Chongqing University of Posts and Telecommunications（重庆邮电大学计算机科学与技术学院）；Intelligent System Department, Zhongxing Telecom Equipment (ZTE)（中兴电信设备智能系统部）
Beihang University（北京航空航天大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）
One of the key issues of Visual Question Answering (VQA) is to reason with semantic clues in the visual content under the guidance of the question, how to model relational semantics still remains as a great challenge. To fully capture visual semantics, we propose to reason over a structured visual representation - scene graph, with embedded objects and inter-object relationships. This shows great benefit over vanilla vector representations and implicit visual relationship learning. Based on existing visual relationship models, we propose a visual relationship encoder that projects visual relat…
Benchmarking Large Multimodal Models for Ophthalmic Visual Question Answering with OphthalWeChat
The Hong Kong Polytechnic University（香港理工大学）；Swiss Federal Institute of Technology Lausanne (EPFL)（洛桑联邦理工学院）；PolyU-Wuhan Technology and Innovation Research Institute（香港理工大学武汉技术创新研究院）；University of Liverpool（利物浦大学）；Centre for Eye and Vision Research (CEVR)（眼科视觉研究中心）
In this work, we propose a deep neural architecture that uses an attention mechanism which utilizes region based image features, the natural language question asked, and semantic knowledge extracted from the regions of an image to produce open-ended answers for questions asked in a visual question answering (VQA) task. The combination of both region based features and region based textual information about the image bolsters a model to more accurately respond to questions and potentially do so with less required training data. We evaluate our proposed architecture on a VQA task against a stron…
Video Question Answering is a challenging problem in visual information retrieval, which provides the answer to the referenced video content according to the question. However, the existing visual question answering approaches mainly tackle the problem of static image question, which may be ineffectively for video question answering due to the insufficiency of modeling the temporal dynamics of video contents. In this paper, we study the problem of video question answering by modeling its temporal dynamics with frame-level attention mechanism. We propose the attribute-augmented attention networ…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Progress in language and image understanding by machines has sparkled the interest of the research community in more open-ended, holistic tasks, and refueled an old AI dream of building intelligent machines. We discuss a few prominent challenges that characterize such holistic tasks and argue for "question answering about images" as a particular appealing instance of such a holistic task. In particular, we point out that it is a version of a Turing Test that is likely to be more robust to over-interpretations and contrast it with tasks like grounding and generation of descriptions. Finally, we…
DermaBench: A Clinician-Annotated Benchmark Dataset for Dermatology Visual Question Answering and Reasoning
Imperial College London（帝国理工学院伦敦分校）；Istanbul Medeniyet University（伊斯坦布尔医学大学）；Usak Research and Training Hospital（Usak研究与培训医院）；Istanbul Research and Training Hospital（伊斯坦布尔研究与培训医院）；Ipswich Hospital（伊普斯韦奇医院）；Medicana Atakoy Hospital（Medicana阿塔基医院）
Large Language Models in the Clinic: A Comprehensive Benchmark
University of Oxford（牛津大学）；Amazon（亚马逊公司）；Harvard T.H. Chan School of Public Health（哈佛陈曾熙公共卫生学院）；Institut polytechnique de Paris（巴黎综合理工学院）；Oxford-Suzhou Centre for Advanced Research（牛津-苏州高等研究院）
MedZERO: Self-Evolving Agents for Open-Ended Medical Reasoning Through Controlled Knowledge Accumulation
The Chinese University of Hong Kong（香港中文大学）；Zhongshan Hospital, Fudan University（复旦大学附属中山医院）；South China University of Technology（华南理工大学）；Chinese Academy of Sciences（中国科学院）
While there has been substantial progress in factoid question-answering (QA), answering complex questions remains challenging, typically requiring both a large body of knowledge and inference techniques. Open Information Extraction (Open IE) provides a way to generate semi-structured knowledge for QA, but to date such knowledge has only been used to answer simple questions with retrieval-based methods. We overcome this limitation by presenting a method for reasoning with Open IE knowledge, allowing more complex questions to be handled. Using a recently proposed support graph optimization frame…
University of Cambridge（剑桥大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Multihop Question Answering is a complex Natural Language Processing task that requires multiple steps of reasoning to find the correct answer to a given question. Previous research has explored the use of models based on Graph Neural Networks for tackling this task. Various architectures have been proposed, including Relational Graph Convolutional Networks (RGCN). For these many node types and relations between them have been introduced, such as simple entity co-occurrences, modelling coreferences, or "reasoning paths" from questions to answers via intermediary entities. Nevertheless, a thoug…
