VLMT：面向多模态多跳问答的视觉-语言多模态Transformer
VLMT: Vision-Language Multimodal Transformer for Multimodal Multi-hop Question Answering
AI总结：
本文提出VLMT统一架构，通过token级视觉-文本注入和三阶段预训练增强跨模态对齐，并构建两阶段MMQA框架，在MultimodalQA和WebQA上显著超越先前最优模型。
摘要：
文本、表格和图像等多模态数据的日益普及，给开发能够进行复杂跨模态推理的模型带来了新的挑战。现有的多模态多跳问答（MMQA）方法往往存在推理能力有限、依赖模态转换以及视觉与文本表示之间对齐不足等问题。为解决这些局限，本文提出视觉-语言多模态Transformer（VLMT），这是一种统一架构，将基于Transformer的视觉编码器与序列到序列语言模型集成在一起。VLMT采用直接的token级注入机制，在共享嵌入空间内融合视觉和文本输入，从而无需中间投影层。为增强跨模态对齐和推理能力，本文提出一种三阶段预训练策略，逐步对齐视觉-语言表示并提升模型的多模态理解能力。基于预训练骨干网络，实例化两个任务特定模块，构成两阶段MMQA框架：一个多模态重排序器，用于预测文档相关性分数，并利用相对阈值与top-k策略进行上下文检索；以及一个多模态问答模型，基于检索到的证据生成有上下文依据的答案。在两个基准数据集上的综合实验证明了所提方法的有效性。在MultimodalQA验证集上，VLMT-Large达到76.5%的精确匹配率和80.1%的F1值，相比此前最先进模型在精确匹配率上提升+9.1%，在F1上提升+8.8%。在WebQA上，其QA得分为47.6，超过PERQA等先前模型+3.2。这些结果凸显了VLMT在多模态推理方面的强大能力，以及其在推进现实世界信息检索和问答系统方面的潜力。
英文摘要：
The increasing availability of multimodal data across text, tables, and images presents new challenges for developing models capable of complex cross-modal reasoning. Existing methods for Multimodal Multi-hop Question Answering (MMQA) often suffer from limited reasoning capabilities, reliance on modality conversion, and inadequate alignment between visual and textual representations. To address these limitations, this paper introduces Vision-Language Multimodal Transformer (VLMT), a unified architecture that integrates a transformer-based vision encoder with a sequence-to-sequence language model. VLMT employs a direct token-level injection mechanism to fuse visual and textual inputs within a shared embedding space, eliminating the need for intermediate projection layers. To enhance cross-modal alignment and reasoning, a three-stage pretraining strategy is proposed to progressively align vision-language representations and improve the model's capacity for multimodal understanding. Based on the pretrained backbone, two task-specific modules are instantiated to form a two-stage MMQA framework: a multimodal reranker that predicts document relevance scores and utilizes a relative threshold with top-k strategy for context retrieval, and a multimodal question answering model that generates contextually grounded answers based on the retrieved evidence. Comprehensive experiments on two benchmark datasets demonstrate the effectiveness of the proposed approach. On MultimodalQA validation set, VLMT-Large achieves 76.5% Exact Match and 80.1% F1, outperforming the previous state-of-the-art by +9.1% in Exact Match and +8.8% in F1. On WebQA, it attains a QA score of 47.6, surpassing prior models such as PERQA by +3.2. These results highlight VLMT's strong capabilities in multimodal reasoning and its potential to advance real-world information retrieval and question answering systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
