ManyModalQA：不同输入下的模态消歧和QA
ManyModalQA: Modality Disambiguation and QA over Diverse Inputs
摘要：
我们提出了一个新的多模态问答的挑战，ManyModalQA，其中代理必须回答一个问题，考虑三个不同的方式：文本，图像和表格。我们通过抓取维基百科来收集数据，然后利用众包来收集问答对。我们的问题是模棱两可的，因为包含答案的模态不容易仅仅根据问题来确定。为了证明这种模糊性，我们构建了一个模态选择器（或消歧器）网络，与现有的数据集相比，这个模型在我们的挑战集上的准确性要低得多，这表明我们的问题更加模糊。通过分析这个模型，我们研究了问题中哪些词是情态指示词。接下来，我们构建一个简单的基线ManyModalQA模型，该模型基于模态选择器的预测，触发相应的预训练的最先进的单峰QA模型。我们专注于为社区提供一个新的多模态评估集，只提供一个微调集，期望现有的数据集和方法将被转移到大部分训练中，以鼓励低资源泛化，而无需为每个新任务提供大型的单一训练集。我们的基线模型和人类表现之间存在显著差距;因此，我们希望这一挑战能够鼓励端到端模态消歧和多模态QA模型以及迁移学习的研究。代码和数据可在https://github.com/hannandarryl/ManyModalQA上获得
英文摘要：
We present a new multimodal question answering challenge, ManyModalQA, in which an agent must answer a question by considering three distinct modalities: text, images, and tables. We collect our data by scraping Wikipedia and then utilize crowdsourcing to collect question-answer pairs. Our questions are ambiguous, in that the modality that contains the answer is not easily determined based solely upon the question. To demonstrate this ambiguity, we construct a modality selector (or disambiguator) network, and this model gets substantially lower accuracy on our challenge set, compared to existing datasets, indicating that our questions are more ambiguous. By analyzing this model, we investigate which words in the question are indicative of the modality. Next, we construct a simple baseline ManyModalQA model, which, based on the prediction from the modality selector, fires a corresponding pre-trained state-of-the-art unimodal QA model. We focus on providing the community with a new manymodal evaluation set and only provide a fine-tuning set, with the expectation that existing datasets and approaches will be transferred for most of the training, to encourage low-resource generalization without large, monolithic training sets for each new task. There is a significant gap between our baseline models and human performance; therefore, we hope that this challenge encourages research in end-to-end modality disambiguation and multimodal QA models, as well as transfer learning. Code and data available at: https://github.com/hannandarryl/ManyModalQA
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
