探索视觉语言模型在帮助视障和低视力人士导航中的应用
Exploring the Use of VLMs for Navigation Assistance for People with Blindness and Low Vision
AI总结：
本文评估了多种VLMs在基础视觉技能和导航任务中的表现，发现GPT-4o在空间推理和场景理解上表现最佳，而开源模型在复杂环境中存在不足，需进一步改进以提升实用性。
摘要：
本文研究了视觉语言模型（VLMs）在帮助视障和低视力人士（pBLV）导航任务中的潜力。我们评估了最先进的闭源模型，包括GPT-4V、GPT-4o、Gemini-1.5-Pro和Claude-3.5-Sonnet，以及开源模型，如Llava-v1.6-mistral和Llava-onevision-qwen，以分析其在基础视觉技能（如计数环境障碍物、相对空间推理和常识性导航-相关场景理解）中的能力。我们进一步评估了它们在导航场景中的表现，使用针对pBLV特定任务设计的提示来模拟现实世界辅助任务。我们的发现揭示了这些模型之间显著的性能差异：GPT-4o在所有任务中均表现最佳，特别是在空间推理和场景理解方面。相比之下，开源模型在复杂环境中的细思熟虑和适应性方面存在困难。常见的挑战包括在拥挤环境中准确计数物体的困难、空间推理中的偏见，以及倾向于优先考虑物体细节而非空间反馈，限制了它们在导航任务中对pBLV的实用性。尽管存在这些限制，VLMs在通过更好地对齐人类反馈和配备改进的空间推理能力时，仍显示出在导航辅助方面的潜力。本研究提供了关于当前VLMs优势和局限性的可操作见解，指导开发者有效将VLMs整合到辅助技术中，同时解决关键限制以提升可用性。
英文摘要：
This paper investigates the potential of vision-language models (VLMs) to assist people with blindness and low vision (pBLV) in navigation tasks. We evaluate state-of-the-art closed-source models, including GPT-4V, GPT-4o, Gemini-1.5-Pro, and Claude-3.5-Sonnet, alongside open-source models, such as Llava-v1.6-mistral and Llava-onevision-qwen, to analyze their capabilities in foundational visual skills: counting ambient obstacles, relative spatial reasoning, and common-sense wayfinding-pertinent scene understanding. We further assess their performance in navigation scenarios, using pBLV-specific prompts designed to simulate real-world assistance tasks. Our findings reveal notable performance disparities between these models: GPT-4o consistently outperforms others across all tasks, particularly in spatial reasoning and scene understanding. In contrast, open-source models struggle with nuanced reasoning and adaptability in complex environments. Common challenges include difficulties in accurately counting objects in cluttered settings, biases in spatial reasoning, and a tendency to prioritize object details over spatial feedback, limiting their usability for pBLV in navigation tasks. Despite these limitations, VLMs show promise for wayfinding assistance when better aligned with human feedback and equipped with improved spatial reasoning. This research provides actionable insights into the strengths and limitations of current VLMs, guiding developers on effectively integrating VLMs into assistive technologies while addressing key limitations for enhanced usability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
