ObjectFinder：面向盲人交互式物体搜索的开放词汇辅助系统
ObjectFinder: An Open-Vocabulary Assistive System for Interactive Object Search by Blind People
AI总结：
提出开放词汇可穿戴辅助系统ObjectFinder，结合开放词汇物体检测器与多模态大语言模型，支持盲人灵活查询、实时定位及多分支信息收集，用户研究证实其在物体搜索中提升了独立性与体验。
摘要：
对于盲人来说，在不熟悉的场景中搜索物体是一项具有挑战性的任务。这涉及指定目标物体、检测它，然后根据用户的意图收集详细信息。然而，现有的基于描述和检测的辅助技术不足以支持交互式物体搜索任务的多面性。我们提出了ObjectFinder，一种开放词汇的可穿戴辅助系统，供盲人进行交互式物体搜索。ObjectFinder允许用户使用灵活的措辞查询目标物体。一旦检测到目标物体，它会实时提供自我中心的定位信息，包括距离和方向。用户随后可以根据对目标物体的意图启动不同的分支以收集详细信息，例如导航至该物体或感知其周围环境。ObjectFinder由开放词汇模型的无缝组合驱动，即一个开放词汇物体检测器和一个多模态大语言模型。ObjectFinder的设计理念及其开发是与一位盲人共同设计师合作完成的。为了评估ObjectFinder，我们与八名盲人参与者进行了一项探索性用户研究。我们将ObjectFinder与BeMyAI和Google Lookout进行了比较，后者是流行的基于描述和检测的辅助应用程序。我们的研究结果表明，大多数参与者在使用ObjectFinder时感到更加独立，并倾向于将其用于物体搜索，因为它增强了场景上下文收集和导航，并允许主动识别目标。最后，我们讨论了未来辅助系统支持交互式物体搜索的意义。
英文摘要：
Searching for objects in unfamiliar scenarios is a challenging task for blind people. It involves specifying the target object, detecting it, and then gathering detailed information according to the user's intent. However, existing description- and detection-based assistive technologies do not sufficiently support the multifaceted nature of interactive object search tasks. We present ObjectFinder, an open-vocabulary wearable assistive system for interactive object search by blind people. ObjectFinder allows users to query target objects using flexible wording. Once the target object is detected, it provides egocentric localization information in real-time, including distance and direction. Users can then initiate different branches to gather detailed information based on their intent towards the target object, such as navigating to it or perceiving its surroundings. ObjectFinder is powered by a seamless combination of open-vocabulary models, namely an open-vocabulary object detector and a multimodal large language model. The ObjectFinder design concept and its development were carried out in collaboration with a blind co-designer. To evaluate ObjectFinder, we conducted an exploratory user study with eight blind participants. We compared ObjectFinder to BeMyAI and Google Lookout, popular description- and detection-based assistive applications. Our findings indicate that most participants felt more independent with ObjectFinder and preferred it for object search, as it enhanced scene context gathering and navigation, and allowed for active target identification. Finally, we discuss the implications for future assistive systems to support interactive object search.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
