一个多模态基础模型，用于辅助盲人和低视力人群进行环境交互
A Multi-Modal Foundation Model to Assist People with Blindness and Low Vision in Environmental Interaction
AI总结：
本文提出一种基于大型视觉语言模型（如InstructBLIP）的多模态方法，通过结合图像标签模型（RAM）与提示工程，为盲人和低视力人群提供环境描述与潜在风险警告，实验证明其能准确识别物体并生成深刻的环境分析。
摘要：
盲人和低视力人群（pBLV）在陌生环境中进行全面的场景识别和精确的物体识别时面临重大挑战。此外，由于视力丧失，pBLV难以自行发现和识别潜在的绊倒危险。本文提出一种开创性方法，利用大型视觉语言模型增强pBLV的视觉感知，提供周围环境的详细全面描述，并对潜在风险发出警告。该方法首先利用大型图像标签模型（即Recognize Anything (RAM)）识别拍摄图像中存在的所有常见物体。随后，将识别结果和用户查询整合到一个提示中，通过提示工程专为pBLV定制。通过结合提示和输入图像，大型视觉语言模型（即InstructBLIP）生成环境的详细全面描述，并通过分析与提示相关的环境物体和场景来识别环境中的潜在风险。我们在室内和室外数据集上进行了实验评估。结果表明，该方法能够准确识别物体，并为pBLV提供深刻的环境描述和分析。
英文摘要：
People with blindness and low vision (pBLV) encounter substantial challenges when it comes to comprehensive scene recognition and precise object identification in unfamiliar environments. Additionally, due to the vision loss, pBLV have difficulty in accessing and identifying potential tripping hazards on their own. In this paper, we present a pioneering approach that leverages a large vision-language model to enhance visual perception for pBLV, offering detailed and comprehensive descriptions of the surrounding environments and providing warnings about the potential risks. Our method begins by leveraging a large image tagging model (i.e., Recognize Anything (RAM)) to identify all common objects present in the captured images. The recognition results and user query are then integrated into a prompt, tailored specifically for pBLV using prompt engineering. By combining the prompt and input image, a large vision-language model (i.e., InstructBLIP) generates detailed and comprehensive descriptions of the environment and identifies potential risks in the environment by analyzing the environmental objects and scenes, relevant to the prompt. We evaluate our approach through experiments conducted on both indoor and outdoor datasets. Our results demonstrate that our method is able to recognize objects accurately and provide insightful descriptions and analysis of the environment for pBLV.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
