arXiv 2412.167712024-12-21cs.CV

SilVar：用于推理视觉问答与目标定位的语音驱动多模态模型

SilVar: Speech Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization

Florida Institute of Technology（佛罗里达理工学院）；FPT University（FPT大学）；Uppsala University（乌普萨拉大学）；Knovel Engineering Lab（诺弗尔工程实验室）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）

Tan-Hanh Pham, Hoang-Nam Le, Phu-Vinh Nguyen, Chris Ngo, Truong-Son Hy

备注：: 10 pages

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出基于CLIP、Whisper和LLaMA 3.1-8B的端到端多模态模型SilVar，利用语音指令进行视觉问答推理与目标定位，并在MMMU和ScienceQA基准上取得SOTA性能。

摘要：

视觉语言模型在视觉问答和图像描述等任务中展现了卓越的能力。然而，大多数模型依赖基于文本的指令，限制了其在人机交互中的效果。此外，语言模型的质量取决于推理和提示技术，如COT，这些在使用语音指令时仍有待深入研究。为应对这些挑战，我们提出了SilVar，一种新颖的端到端多模态模型，利用语音指令进行视觉问答推理。此外，我们研究了不同层级的推理技术，包括对话式、简单和复杂的语音指令。SilVar基于CLIP、Whisper和LLaMA 3.1-8B构建，允许用户提供语音或文本指令，实现直观交互。为此，我们引入了一个数据集，旨在通过基于语音的推理任务挑战模型的目标定位能力。该数据集增强了模型从语音输入中处理和解释视觉场景的能力，使其从目标识别迈向基于推理的交互。实验表明，尽管面临语音指令的挑战，SilVar仍在MMMU和ScienceQA基准上取得了SOTA性能。我们相信SilVar将启发下一代多模态推理模型，迈向专家级通用人工智能。我们的代码和数据集已在此处提供。

英文摘要：

Visual Language Models have demonstrated remarkable capabilities across tasks, including visual question answering and image captioning. However, most models rely on text-based instructions, limiting their effectiveness in human-machine interactions. Moreover, the quality of language models depends on reasoning and prompting techniques, such as COT, which remain underexplored when using speech instructions. To address these challenges, we propose SilVar, a novel end-to-end multimodal model that uses speech instructions for reasoning in visual question answering. In addition, we investigate reasoning techniques with levels including conversational, simple, and complex speech instruction. SilVar is built upon CLIP, Whisper, and LLaMA 3.1-8B, enabling intuitive interactions by allowing users to provide verbal or text instructions. To this end, we introduce a dataset designed to challenge models with speech-based reasoning tasks for object localization. This dataset enhances the model ability to process and explain visual scenes from spoken input, moving beyond object recognition to reasoning-based interactions. The experiments show that SilVar achieves SOTA performance on the MMMU and ScienceQA benchmarks despite the challenge of speech-based instructions. We believe SilVar will inspire next-generation multimodal reasoning models, toward expert artificial general intelligence. Our code and dataset are available here.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。