Open-domain extractive question answering works well on textual data by first retrieving candidate texts and then extracting the answer from those candidates. However, some questions cannot be answered by text alone but require information stored in tables. In this paper, we present an approach for retrieving both texts and tables relevant to a question by jointly encoding texts, tables and questions into a single vector space. To this end, we create a new multi-modal dataset based on text and table datasets from related work and compare the retrieval performance of different encoding schemata. We find that dense vector embeddings of transformer models outperform sparse embeddings on four out of six evaluation datasets. Comparing different dense embedding models, tri-encoders with one encoder for each question, text and table, increase retrieval performance compared to bi-encoders with one encoder for the question and one for both text and tables. We release the newly created multi-modal dataset to the community so that it can be used for training and evaluation.
MultiModalQA: Complex Question Answering over Text, Tables and Images
The Allen Institute for AI（艾伦人工智能研究所）；Tel-Aviv University（特拉维夫大学）；University of Washington（华盛顿大学）
在回答复杂的问题时，人们可以无缝地结合来自视觉，文本和表格来源的信息。虽然近年来对多个证据推理的模型的兴趣激增，但对跨多个模态推理的问答模型的研究相对较少。在本文中，我们提出了MultiModalQA（MMQA）：一个具有挑战性的问答数据集，需要联合推理的文本，表格和图像。我们使用一个新的框架来创建MMQA，该框架用于大规模生成复杂的多模态问题，从维基百科获取表格，并使用每个表格中出现的实体附加图像和文本段落。然后，我们定义了一个形式语言，使我们能够采取的问题，可以回答从一个单一的模态，并将它们结合起来，以产生跨模态的问题。最后，众包工作人员将这些自动生成的问题转化为更流畅的语言。我们通过这个过程创建了29，918个问题，并以经验证明了多模态多跳方法解决我们任务的必要性：我们的多跳模型ImplicitDecomp在跨模态问题上的平均F1为51.7，大大超过了达到38.2 F1的强大基线，但仍然明显落后于人类表现，这是90.1 F1
EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images
KAIST（韩国科学技术院）；Microsoft Research Asia（微软亚洲研究院）；Centre of Perceptual Interactive Intelligence（感知交互智能中心）；Seoul National University Bundang Hospital（首尔大学盆唐医院）
SciMMIR: Benchmarking Scientific Multi-modal Information Retrieval
Multimodal Art Projection Research Community（多模态艺术投影研究共同体）；University of Manchester（曼彻斯特大学）；University of Waterloo（滑铁卢大学）；Hong Kong University of Science and Technology（香港科技大学）；Durham University（杜伦大学）
HIPPO: Enhancing the Table Understanding Capability of LLMs through Hybrid-Modal Preference Optimization
School of Computer Science and Engineering, Northeastern University, Shenyang, China（东北大学计算机科学与工程学院）；Department of Computer Science and Technology, Tsinghua University, Beijing, China（清华大学计算机科学与技术系）；Huawei Technologies Co., Ltd（华为技术有限公司）
Bridge the Gap between Language models and Tabular Understanding
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；Microsoft（微软）；Duke University（杜克大学）；Yale University（耶鲁大学）；Peking University（北京大学）
Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning
State Key Laboratory of Novel Software Technology, Nanjing University, China（新型软件技术国家重点实验室，南京大学，中国）；School of Artificial Intelligence, Nanjing University, China（人工智能学院，南京大学，中国）；School of Intelligence Science and Technology, Nanjing University, China（智能科学与技术学院，南京大学，中国）
All Data on the Table: Novel Dataset and Benchmark for Cross-Modality Scientific Information Extraction
Nankai University（南开大学）；Tokyo Institute of Technology（东京工业大学）；Microsoft Research Asia（微软亚洲研究院）；Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院（BAAI））
Table Understanding and (Multimodal) LLMs: A Cross-Domain Case Study on Scientific vs. Non-Scientific Data
Deutsches Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)（德国人工智能研究中心有限公司）；Technische Universität Berlin（柏林技术大学）；BIFOLD；Deutsche Telekom（德国电信）；Common Crawl Foundation（Common Crawl基金会）；Humboldt-Universität zu Berlin（柏林洪堡大学）
