利用盲人和低视力人群视觉问题引导多模态大语言模型实现主动视觉解读
Guiding Multimodal Large Language Models with Blind and Low Vision People Visual Questions for Proactive Visual Interpretations
AI总结：
针对现有多模态大语言模型视觉解读应用默认生成冗长无关描述的问题，研究人员开发了借鉴BLV用户历史问题的系统，从VizWiz-LF数据集匹配相似语境引导生成相关描述，经评估效果更优且代码开源。
摘要：
多模态大语言模型（MLLMs）凭借其准确性及提供丰富类人解读的能力，已被整合到视觉解读应用中，为盲人和低视力（BLV）用户提供支持。然而，这类应用通常默认生成全面冗长的描述，不考虑具体语境，导致交互效率低下——用户不得不浏览无关细节，而非获取他们可能想要查找的特定信息。为提供更具语境相关性的信息，我们开发了一套借鉴BLV用户历史问题的系统。给定一张图像时，该系统从VizWiz-LF数据集中识别相似的过往视觉语境，并利用关联问题引导MLLM生成与BLV用户更相关的描述。由三名人类标注员对92条具备语境感知和无语境的描述进行修订评估，结果显示语境感知描述在76.1%的案例（92例中的70例）中预判并回答了用户问题，在54.4%的对比（92例中的50例）更受偏好。我们的论文综述及数据分析已在GitHub仓库公开，地址为https://github.com/rgonzalezp/guiding-multimodal-large-language-models-with-blind-and-low-vision-people-visual-questions 。
英文摘要：
Multimodal large language models (MLLMs) have been integrated into visual interpretation applications to support Blind and Low Vision (BLV) users because of their accuracy and ability to provide rich, human-like interpretations. However, these applications often default to comprehensive, lengthy descriptions regardless of context. This leads to inefficient exchanges, as users must go through irrelevant details rather than receiving the specific information they are likely to seek. To deliver more contextually-relevant information, we developed a system that draws on historical BLV users questions. When given an image, our system identifies similar past visual contexts from the VizWiz-LF dataset and uses the associated questions to guide the MLLM generate descriptions more relevant to BLV users. An evaluation with three human labelers who revised 92 context-aware and context-free descriptions showed that context-aware descriptions anticipated and answered users' questions in 76.1% of cases (70 out of 92) and were preferred in 54.4% of comparisons (50 out of 92). Our paper reviews, and data analysis are publicly available in a Github repository at https://github.com/rgonzalezp/guiding-multimodal-large-language-models-with-blind-and-low-vision-people-visual-questions .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
