利用合成任务数据评估多模态推理模型的能力
Evaluating the Capabilities of Multi-modal Reasoning Models with Synthetic Task Data
AI总结：
本文提出基于高分辨率文本到图像生成的多模态推理评估数据合成框架，构建上下文相关异常检测数据集，并揭示SOTA VQA模型在该任务上显著弱于标准VQA。
摘要：
大型语言模型以及语言与视觉联合理解模型所取得的显著进展和应用，使人们越来越需要用于探测其潜在推理能力的方法。然而，为复杂的多模态推理任务收集自然出现的数据十分困难，这限制了对学术数据集尚未覆盖任务上的AI方法进行评估。在本工作中，我们利用高分辨率文本到图像生成的最新进展，开发了一个用于生成多模态推理任务评估数据的框架。我们应用该框架生成依赖上下文的异常数据，并在一个现有数据集未能很好覆盖的具有挑战性的任务上创建了合成数据集。我们将最先进的视觉问答（VQA）模型在用该方法生成的数据上进行性能基准测试，并证明尽管该任务是可处理的，但模型在依赖上下文的异常检测任务上的表现明显差于标准VQA任务。
英文摘要：
The impressive advances and applications of large language and joint language-and-visual understanding models has led to an increased need for methods of probing their potential reasoning capabilities. However, the difficulty of gather naturally-occurring data for complex multi-modal reasoning tasks bottlenecks the evaluation of AI methods on tasks which are not already covered by an academic dataset. In this work, we leverage recent advances in high resolution text-to-image generation to develop a framework for generating evaluation data for multi-modal reasoning tasks. We apply this framework to generate context-dependent anomaly data, creating a synthetic dataset on a challenging task which is not well covered by existing datasets. We benchmark the performance of a state-of-the-art visual question answering (VQA) model against data generated with this method, and demonstrate that while the task is tractable, the model performs significantly worse on the context-dependent anomaly detection task than on standard VQA tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
