基于自我中心视觉-语言模型的可便携实时智能助手
An Egocentric Vision-Language Model based Portable Real-time Smart Assistant
AI总结：
提出视觉-语言系统Vinci，通过集成自我中心视觉基础模型与LLM的EgoVideo-VL，结合记忆、生成和检索模块，实现便携设备上的实时场景理解与技能辅助，为自我中心AI系统建立新框架。
摘要：
我们提出了Vinci，一个旨在为便携设备提供实时、全面AI辅助的视觉-语言系统。其核心是利用EgoVideo-VL，这是一种将自我中心视觉基础模型与大型语言模型（LLM）集成的新颖模型，可实现场景理解、时间定位、视频摘要和未来规划等高级功能。为增强实用性，Vinci包含用于实时处理长视频流并保留上下文历史的记忆模块、用于生成视觉动作演示的生成模块，以及连接自我中心与第三人称视角以提供相关教学视频辅助技能获取的检索模块。与依赖专用硬件的现有系统不同，Vinci不依赖特定硬件，支持在智能手机和可穿戴相机等多种设备上部署。在实验中，我们首先展示了EgoVideo-VL在多个公共基准上的卓越性能，突显其视觉-语言推理与上下文理解能力。随后通过用户研究评估Vinci在现实场景的有效性，强调其在多样场景中的适应性与可用性。我们希望Vinci能为便携式实时自我中心AI系统建立新框架，为用户提供上下文相关且可操作的见解。Vinci的前端、后端及模型代码均发布于https://github.com/OpenGVLab/vinci。
英文摘要：
We present Vinci, a vision-language system designed to provide real-time, comprehensive AI assistance on portable devices. At its core, Vinci leverages EgoVideo-VL, a novel model that integrates an egocentric vision foundation model with a large language model (LLM), enabling advanced functionalities such as scene understanding, temporal grounding, video summarization, and future planning. To enhance its utility, Vinci incorporates a memory module for processing long video streams in real time while retaining contextual history, a generation module for producing visual action demonstrations, and a retrieval module that bridges egocentric and third-person perspectives to provide relevant how-to videos for skill acquisition. Unlike existing systems that often depend on specialized hardware, Vinci is hardware-agnostic, supporting deployment across a wide range of devices, including smartphones and wearable cameras. In our experiments, we first demonstrate the superior performance of EgoVideo-VL on multiple public benchmarks, showcasing its vision-language reasoning and contextual understanding capabilities. We then conduct a series of user studies to evaluate the real-world effectiveness of Vinci, highlighting its adaptability and usability in diverse scenarios. We hope Vinci can establish a new framework for portable, real-time egocentric AI systems, empowering users with contextual and actionable insights. Including the frontend, backend, and models, all codes of Vinci are available at https://github.com/OpenGVLab/vinci.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
