arXiv 2503.038032026-02-09cs.CV

EgoLife：迈向以自身为中心的生活助手

EgoLife: Towards Egocentric Life Assistant

S-Lab, Nanyang Technological University, Singapore（南洋理工大学新加坡分校）；LMMs-Lab（LMMs实验室）；IMDEA Networks, Spain（西班牙IMDEA网络）；University of Brescia, Italy（意大利布雷西亚大学）；Politecnico di Milano, Italy（意大利米兰理工学院）

Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, Bei Ouyang, Zhengyu Lin,… 展开作者

Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, Bei Ouyang, Zhengyu Lin, Marco Cominelli, Zhongang Cai, Yuanhan Zhang, Peiyuan Zhang, Fangzhou Hong, Joerg Widmer, Francesco Gringoli, Lei Yang, Bo Li, Ziwei Liu

备注：: This version corrects the author affiliation to reflect the accurate institutional information at the time of publication. No technical content of the paper has been changed

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

EgoLife旨在通过AI赋能的可穿戴设备开发以自身为中心的生活助手，通过多模态数据集和EgoButler系统提升日常效率与个性化服务。

摘要：

我们介绍了EgoLife，一个开发以自身为中心的生活助手的项目，通过AI赋能的可穿戴眼镜来陪伴并提升个人效率。为了为这个助手奠定基础，我们进行了全面的数据收集研究，其中六位参与者共同生活了一周，持续记录他们的日常活动——包括讨论、购物、烹饪、社交和娱乐——使用AI眼镜进行多模态的以自身为中心的视频采集，同时同步录制第三方视角的视频参考。这项努力产生了EgoLife数据集，这是一个全面的300小时以自身为中心、人际互动、多视角和多模态的日常生活数据集，具有密集标注。利用这个数据集，我们介绍了EgoLifeQA，一套长上下文、生活导向的问题回答任务，旨在通过回答诸如回忆相关事件、监测健康习惯和提供个性化推荐等实际问题来提供有意义的帮助。为了解决（1）开发稳健的视觉-音频模型以处理以自身为中心的数据、（2）实现身份识别以及（3）在大量时间信息上实现长上下文问题回答的关键技术挑战，我们介绍了EgoButler，一个集成了EgoGPT和EgoRAG的系统。EgoGPT是一个在以自身为中心的数据集上训练的多模态模型，实现了在以自身为中心的视频理解上的最新性能。EgoRAG是一个基于检索的组件，支持回答超长上下文的问题。我们的实验研究验证了它们的工作机制，并揭示了关键因素和瓶颈，指导未来改进。通过发布我们的数据集、模型和基准，我们旨在激发进一步研究以自身为中心的AI助手。

英文摘要：

We introduce EgoLife, a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses. To lay the foundation for this assistant, we conducted a comprehensive data collection study where six participants lived together for one week, continuously recording their daily activities - including discussions, shopping, cooking, socializing, and entertainment - using AI glasses for multimodal egocentric video capture, along with synchronized third-person-view video references. This effort resulted in the EgoLife Dataset, a comprehensive 300-hour egocentric, interpersonal, multiview, and multimodal daily life dataset with intensive annotation. Leveraging this dataset, we introduce EgoLifeQA, a suite of long-context, life-oriented question-answering tasks designed to provide meaningful assistance in daily life by addressing practical questions such as recalling past relevant events, monitoring health habits, and offering personalized recommendations. To address the key technical challenges of (1) developing robust visual-audio models for egocentric data, (2) enabling identity recognition, and (3) facilitating long-context question answering over extensive temporal information, we introduce EgoButler, an integrated system comprising EgoGPT and EgoRAG. EgoGPT is an omni-modal model trained on egocentric datasets, achieving state-of-the-art performance on egocentric video understanding. EgoRAG is a retrieval-based component that supports answering ultra-long-context questions. Our experimental studies verify their working mechanisms and reveal critical factors and bottlenecks, guiding future improvements. By releasing our datasets, models, and benchmarks, we aim to stimulate further research in egocentric AI assistants.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。