SUPERGLASSES：基于智能眼镜的视觉语言模型智能体基准测试
SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glasses
AI总结：
本文提出SUPERGLASSES基准，通过真实世界数据评估智能眼镜的多模态交互能力，提出SUPERLENS模型在VQA任务中超越GPT-4o，揭示了任务特定解决方案的重要性。
摘要：
AI赋能的智能眼镜作为最热的可穿戴设备之一，已解锁了多模态交互的新前沿，视觉问答（VQA）在外部知识源上的应用成为核心。现有视觉语言模型（VLMs）适应智能眼镜通常是在传统多模态数据集上训练和评估；然而，这些数据集缺乏智能眼镜使用场景所需的多样性和现实性，并与特定挑战相去甚远，其中准确识别感兴趣对象必须在任何外部知识检索之前。为弥合这一差距，我们引入SUPERGLASSES，这是首个基于真实世界数据构建的VQA基准，完全由智能眼镜设备收集。SUPERGLASSES包含2,422个第一人称视角图像-问题对，涵盖14个图像领域和8个查询类别，附有完整的搜索轨迹和推理注释。我们在该基准上评估了26个代表性VLMs，揭示了显著的性能差距。为了解决现有模型的局限性，我们进一步提出SUPERLENS，一个多模态智能眼镜代理，通过整合自动对象检测、查询解耦和多模态网络搜索，实现检索增强的答案生成。SUPERLENS在VQA任务中取得最佳性能，超越GPT-4o 2.19%，突显了智能眼镜VQA中任务特定解决方案的必要性。我们的数据集在https://huggingface.co/datasets/xandery/SuperGlasses公开可用。
英文摘要：
The rapid advancement of AI-powered smart glasses-one of the hottest wearable devices-has unlocked new frontiers for multimodal interaction, with Visual Question Answering (VQA) over external knowledge sources emerging as a core application. Existing Vision Language Models (VLMs) adapted to smart glasses are typically trained and evaluated on traditional multimodal datasets; however, these datasets lack the variety and realism needed to reflect smart glasses usage scenarios and diverge from their specific challenges, where accurately identifying the object of interest must precede any external knowledge retrieval. To bridge this gap, we introduce SUPER- GLASSES, the first comprehensive VQA benchmark built on real-world data entirely collected by smart glasses devices. SUPERGLASSES comprises 2,422 egocentric image-question pairs spanning 14 image domains and 8 query categories, enriched with full search trajectories and reasoning annotations. We evaluate 26 representative VLMs on this benchmark, revealing significant performance gaps. To address the limitations of existing models, we further propose the SUPERLENS, a multimodal smart glasses agent that enables retrieval-augmented answer generation by integrating automatic object detection, query decoupling, and multimodal web search. SUPERLENS achieves state-of-the-art performance, outperforming GPT-4o by 2.19%, underscoring the need for task-specific solutions in smart glasses VQA. Our dataset is publicly available at https://huggingface.co/datasets/xandery/SuperGlasses.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
