我们到了吗？探索多模态大语言模型在辅助人工智能应用中的能力
Are We There Yet? Exploring the Capabilities of MLLMs in Assistive AI Applications
AI总结：
本文通过真实世界任务评估多模态大语言模型在辅助AI中的表现，开发NetraLink系统收集基准数据，揭示其优势与局限。
摘要：
多模态大语言模型（MLLMs）通过将视觉编码器与大规模语言模型相结合，重新定义了视觉理解。这种统一架构使其在图像描述、视觉问答和多模态对话等任务中表现出色，通常是在零样本或少样本设置下。它们的通用能力和灵活接口使MLLMs成为现实世界视觉语言应用的有前景基础。辅助AI旨在帮助用户通过自然语言与环境交互。这些场景需要强大的视觉识别、上下文推理和多语言理解能力——这些正是MLLMs被认为具备的能力。然而，它们在辅助环境中的有效性仍有待全面理解。在这项工作中，我们通过评估最先进模型在真实世界任务上的表现，探索MLLMs是否能够支持辅助AI：识别日常物品如货币、基于场景文本回答问题、以及跨多种语言阅读视觉呈现的内容。为此，我们开发了一个系统NetraLink，使用头戴式GoPro捕获真实世界的自我中心数据，并收集了一个涵盖这些辅助场景的基准。我们的发现提供了对当前MLLMs的全面诊断，突出了它们在实现基于视觉感知和语言交互的辅助技术方面的优势与局限。
英文摘要：
Multimodal Large Language Models (MLLMs) have redefined visual understanding by combining vision encoders with large-scale language models. This unified architecture enables strong performance on tasks like image captioning, visual question answering, and multimodal dialogue, often in zero- and few-shot settings. Their general-purpose capabilities and flexible interfaces make MLLMs a promising foundation for real-world vision-language applications. Assistive AI aims to help users interact with their environments through natural language. These scenarios demand robust visual recognition, contextual reasoning, and multilingual comprehension-capabilities that MLLMs are believed to offer. However, their effectiveness in assistive settings remains to be fully understood. In this work, we explore whether MLLMs can support Assistive AI by evaluating state-of-the-art models on real-world tasks: recognizing everyday objects like currency, answering questions based on scene text, and reading visually presented content across multiple languages. To this end, we developed a system, NetraLink, using a head-mounted GoPro to capture real-world egocentric data, and collected a benchmark covering these assistive scenarios. Our findings provide a comprehensive diagnostic of current MLLMs, highlighting their strengths and limitations in enabling assistive technologies grounded in visual perception and language interaction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
