X-InstructBLIP：一个将X模态指令感知表示与LLM对齐并实现涌现跨模态推理的框架
X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning
AI总结：
本文提出X-InstructBLIP框架，将图像、3D、音频、视频对齐到冻结LLM，比较Q-Former与线性投影，并构建指令数据及DisCRn跨模态推理基准。
摘要：
近期研究通过学习图像到语言的投影并利用大型语言模型（LLM）令人瞩目的推理能力，在视觉推理任务上取得了显著进展。本文介绍了一个高效且有效的框架，将多种模态（图像、3D、音频和视频）集成到冻结的LLM中，并展示了跨模态推理（2种或更多模态输入）的涌现能力。我们的方法探索了两种不同的投影机制：Q-Formers和线性投影（LPs）。通过在全部四种模态、16个基准上进行大量实验，我们研究了这两种方法，并评估它们在集成和独立跨模态推理中的适应性。Q-Former投影在单模态场景中表现出更优性能，并在涉及两种或更多模态的联合推理与判别推理中展现出适应性。然而，在任务-模态数据有限的情境下，其泛化能力低于线性投影。为实现该框架，我们设计了一条可扩展流水线，可从不同模态中现成可用的字幕数据自动生成高质量的指令微调数据集，并贡献了24K条音频QA数据和250K条3D QA数据。为促进跨模态推理的进一步研究，我们提出了DisCRn（判别式跨模态推理）基准，包含9K个音频-视频QA样本和28K个图像-3D QA样本，要求模型在不同输入模态之间进行判别式推理。
英文摘要：
Recent research has achieved significant advancements in visual reasoning tasks through learning image-to-language projections and leveraging the impressive reasoning abilities of Large Language Models (LLMs). This paper introduces an efficient and effective framework that integrates multiple modalities (images, 3D, audio and video) to a frozen LLM and demonstrates an emergent ability for cross-modal reasoning (2+ modality inputs). Our approach explores two distinct projection mechanisms: Q-Formers and Linear Projections (LPs). Through extensive experimentation across all four modalities on 16 benchmarks, we explore both methods and assess their adaptability in integrated and separate cross-modal reasoning. The Q-Former projection demonstrates superior performance in single modality scenarios and adaptability in joint versus discriminative reasoning involving two or more modalities. However, it exhibits lower generalization capabilities than linear projection in contexts where task-modality data are limited. To enable this framework, we devise a scalable pipeline that automatically generates high-quality, instruction-tuning datasets from readily available captioning data across different modalities, and contribute 24K QA data for audio and 250K QA data for 3D. To facilitate further research in cross-modal reasoning, we introduce the DisCRn (Discriminative Cross-modal Reasoning) benchmark comprising 9K audio-video QA samples and 28K image-3D QA samples that require the model to reason discriminatively across disparate input modalities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
