Voila-A：将视觉-语言模型与用户注视注意力对齐
Voila-A: Aligning Vision-Language Models with User's Gaze Attention
AI总结：
本文提出Voila-A方法，通过引入AR/VR设备采集的注视信息引导视觉-语言模型，并构建VOILA-COCO数据集和Voila Perceiver模块，实现模型注意力与人类注视模式对齐，显著提升真实场景下的模型可解释性与性能。
摘要：
近年来，视觉与语言理解的融合推动了人工智能的重大进步，尤其是通过视觉-语言模型（VLMs）。然而，现有VLMs在处理具有复杂场景和多物体的真实世界应用时面临挑战，也难以将其关注点与人类用户多样化的注意力模式对齐。在本文中，我们引入可通过AR或VR设备便捷采集的注视信息，作为人类注意力的代理来引导VLMs，并提出一种新颖的方法Voila-A，用于注视对齐，以增强这些模型在真实世界应用中的可解释性和有效性。首先，我们采集了数百分钟的注视数据，以证明我们可以使用局部化叙述来模拟人类注视模态。然后，我们设计了一个利用GPT-4的自动数据标注流程，生成VOILA-COCO数据集。此外，我们创新了Voila Perceiver模块，将注视信息整合到VLMs中，同时保留其预训练知识。我们使用留出验证集和新采集的VOILA-GAZE测试集对Voila-A进行评估，该测试集包含通过注视追踪设备捕获的真实生活场景。实验结果表明，Voila-A显著优于多个基线模型。通过将模型注意力与人类注视模式对齐，Voila-A为更直观、以用户为中心的VLMs铺平了道路，并在广泛的应用中促进了有吸引力的人机交互。
英文摘要：
In recent years, the integration of vision and language understanding has led to significant advancements in artificial intelligence, particularly through Vision-Language Models (VLMs). However, existing VLMs face challenges in handling real-world applications with complex scenes and multiple objects, as well as aligning their focus with the diverse attention patterns of human users. In this paper, we introduce gaze information, feasibly collected by AR or VR devices, as a proxy for human attention to guide VLMs and propose a novel approach, Voila-A, for gaze alignment to enhance the interpretability and effectiveness of these models in real-world applications. First, we collect hundreds of minutes of gaze data to demonstrate that we can mimic human gaze modalities using localized narratives. We then design an automatic data annotation pipeline utilizing GPT-4 to generate the VOILA-COCO dataset. Additionally, we innovate the Voila Perceiver modules to integrate gaze information into VLMs while preserving their pretrained knowledge. We evaluate Voila-A using a hold-out validation set and a newly collected VOILA-GAZE Testset, which features real-life scenarios captured with a gaze-tracking device. Our experimental results demonstrate that Voila-A significantly outperforms several baseline models. By aligning model attention with human gaze patterns, Voila-A paves the way for more intuitive, user-centric VLMs and fosters engaging human-AI interaction across a wide range of applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
