GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning
Sun Yat-sen University(中山大学);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Dark Matter AI Inc.(暗物智能科技有限公司);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P. Xing, Liang Lin
Automatic math problem solving has recently attracted increasing attention as a long-standing AI benchmark. In this paper, we focus on solving geometric problems, which requires a comprehensive understanding of textual descriptions, visual diagrams, and theorem knowledge. However, the existing methods were highly dependent on handcraft rules and were merely evaluated on small-scale datasets. Therefore, we propose a Geometric Question Answering dataset GeoQA, containing 4,998 geometric problems with corresponding annotated programs, which illustrate the solving process of the given problems. Compared with another publicly available dataset GeoS, GeoQA is 25 times larger, in which the program annotations can provide a practical testbed for future research on explicit and explainable numerical reasoning. Moreover, we introduce a Neural Geometric Solver (NGS) to address geometric problems by comprehensively parsing multimodal information and generating interpretable programs. We further add multiple self-supervised auxiliary tasks on NGS to enhance cross-modal semantic representation. Extensive experiments on GeoQA validate the effectiveness of our proposed NGS and auxiliary tasks. However, the results are still significantly lower than human performance, which leaves large room for future research. Our benchmark and code are released at https://github.com/chen-judge/GeoQA .
TABQAWORLD: Optimizing Multimodal Reasoning for Multi-Turn Table Question Answering
University of California, Los Angeles(加州大学洛杉矶分校);McGill University(麦吉尔大学);Université de Montréal(蒙特利尔大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));SimpleWay
Beyond Retrieval: Joint Supervision and Multimodal Document Ranking for Textbook Question Answering
Faculty of Computing and Information Technology, King Abdulaziz University(计算机与信息科技学院,国王阿卜杜勒阿齐兹大学);College of Computer and Information Science, Imam Mohammad Ibn Saud Islamic University (IMSIU)(计算机与信息科学学院,伊玛目穆罕默德·本·萨乌德伊斯兰大学);School of Computing, Macquarie University(计算学院,麦考瑞大学)
M$^3$-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering
Interdisciplinary Program in Artificial Intelligence(人工智能交叉学科项目);Department of Intelligence and Information(智能与信息系);Samsung Advanced Institute of Technology(三星先进技术研究院)
Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairs
School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);NiuTrans Research(NiuTrans研究院);School of Software, Northeastern University(东北大学软件学院)
MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answering
Yale University, New Haven, CT, USA(耶鲁大学);McGill University, Montreal, QC, Canada(麦吉尔大学);University of Texas Rio Grande Valley, TX, USA(德克萨斯大学里奥格兰德谷分校)
Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Framework
College of Computer Science, Chongqing University(重庆大学计算机学院);Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院);Kuaishou Technology(快手科技)
ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering
OPPO AI Center, OPPO Inc. China(OPPO AI中心,OPPO公司);The Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Nanyang Technological University, Singapore(新加坡南洋理工大学)
HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering
Human-inspired AI Research, Korea University(韩国大学人机智能研究部);Computer Science and Engineering, Konkuk University(韩国康科大学计算机科学与工程系);Department of Computer Science and Engineering, Korea University(韩国大学计算机科学与工程系)
MultiModalQA: Complex Question Answering over Text, Tables and Images
The Allen Institute for AI(艾伦人工智能研究所);Tel-Aviv University(特拉维夫大学);University of Washington(华盛顿大学)
在回答复杂的问题时,人们可以无缝地结合来自视觉,文本和表格来源的信息。虽然近年来对多个证据推理的模型的兴趣激增,但对跨多个模态推理的问答模型的研究相对较少。在本文中,我们提出了MultiModalQA(MMQA):一个具有挑战性的问答数据集,需要联合推理的文本,表格和图像。我们使用一个新的框架来创建MMQA,该框架用于大规模生成复杂的多模态问题,从维基百科获取表格,并使用每个表格中出现的实体附加图像和文本段落。然后,我们定义了一个形式语言,使我们能够采取的问题,可以回答从一个单一的模态,并将它们结合起来,以产生跨模态的问题。最后,众包工作人员将这些自动生成的问题转化为更流畅的语言。我们通过这个过程创建了29,918个问题,并以经验证明了多模态多跳方法解决我们任务的必要性:我们的多跳模型ImplicitDecomp在跨模态问题上的平均F1为51.7,大大超过了达到38.2 F1的强大基线,但仍然明显落后于人类表现,这是90.1 F1