WalkVLM：利用视觉语言模型辅助视障人士行走
WalkVLM:Aid Visually Impaired People Walking by Vision Language Model
AI总结：
本文提出首个用于视障行走辅助的大规模数据集（含12,000个视频-标注对）及WalkVLM模型，通过思维链分层规划和时间感知自适应预测生成简洁提醒，在流媒体视频处理基准上优于其他VLMs。
摘要：
全球约有2亿人患有不同程度的视力障碍，因此利用AI技术为这些人提供行走辅助至关重要。随着视觉语言模型（VLMs）的近期进展，应用VLMs提供行走指导变得流行。然而，现有的行走指导方法主要基于不可公开访问的自建问答数据集，缺乏用于训练或评估的标准化基准。此外，行走辅助通常需要实时流媒体视频分析并生成简洁而信息丰富的提醒，但由于响应过度和推理效率低下，VLMs在此方面面临困难。在本文中，我们引入了首个专用于行走辅助的大规模数据集，包含12,000个视频-标注对，为训练和评估帮助视障人士行走的系统提供统一基准。此外，提出了WalkVLM模型，该模型采用思维链进行分层规划以生成简洁但信息丰富的提醒，并利用时间感知自适应预测来减少提醒的时间冗余。最后，我们为盲人行走任务建立了坚实的基准，并验证了WalkVLM在此任务的流媒体视频处理中相比其他VLMs的优势。我们的数据集和代码可在 https://walkvlm2024.github.io 获取。
英文摘要：
Approximately 200 million individuals around the world suffer from varying degrees of visual impairment, making it crucial to leverage AI technology to offer walking assistance for these people. With the recent progress of vision-language models (VLMs), applying VLMs to offer walking guidance has become popular. However, the existing methods of walking guidance are mainly based on self-curated question-answering datasets that are not publicly accessible, without a standardized benchmark for training or evaluation. Moreover, walking assistance often requires real-time streaming video analysis and the generation of concise yet informative reminders, making VLMs struggle due to excessive responses and low efficiency in inferences. In this paper, we introduce the first large-scale dataset dedicated to walking assistance, comprising 12,000 video-annotation pairs, to provide a unified benchmark for training and evaluating systems to help visually-impaired individuals walk. Furthermore, a WalkVLM model is proposed, which employs chain of thought for hierarchical planning to generate concise but informative reminders and utilizes temporal-aware adaptive prediction to reduce the temporal redundancy of reminders. Finally, we have established a solid benchmark for blind walking task and verified the advantages of WalkVLM in stream video processing for this task compared to other VLMs. Our dataset and code are available at https://walkvlm2024.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
