构建第一人称程序化AI助手：方法、基准和挑战
Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges
AI总结：
本文提出第一人称程序化AI助手，探讨其核心任务、赋能维度及挑战，通过实验评估现有方法并指明未来研究方向。
摘要：
受近期视觉-语言模型（VLMs）和第一人称感知研究的进展推动，本文引入了新兴的'第一人称程序化AI助手'（EgoProceAssist）这一研究领域，旨在通过第一人称视角逐步支持日常程序性任务。本文首先确定了EgoProceAssist的三个核心任务：第一人称程序性错误检测、第一人称程序性学习和第一人称程序性问答，然后引入两个赋能维度：实时和流媒体视频理解，以及程序性上下文中的主动交互。我们将这些任务定义为新的分类法中的EgoProceAssist基本功能，并说明它们如何在真实世界场景中部署为日常活动助手。具体而言，我们的工作涵盖了当前技术、相关数据集和评估指标在这些五个核心领域的全面回顾。为了澄清所提出的EgoProceAssist与现有基于VLM的助手之间的差距，我们进行了新的实验，以提供对代表性VLM方法的全面评估。通过这些发现和我们的技术分析，我们讨论了未来面临的挑战，并提出了未来的研究方向。此外，本研究的详尽列表已公开在活跃的存储库中，该存储库持续收集最新工作：https://github.com/z1oong/Building-Egocentric-Procedural-AI-Assistant。
英文摘要：
Driven by recent advances in vision-language models (VLMs) and egocentric perception research, the emerging topic of an egocentric procedural AI assistant (EgoProceAssist) is introduced to step-by-step support daily procedural tasks in a first-person view. In this paper, we start by identifying three core tasks in EgoProceAssist: egocentric procedural error detection, egocentric procedural learning, and egocentric procedural question answering, then introduce two enabling dimensions: real-time and streaming video understanding, and proactive interaction in procedural contexts. We define these tasks within a new taxonomy as the EgoProceAssist's essential functions and illustrate how they can be deployed in real-world scenarios for daily activity assistants. Specifically, our work encompasses a comprehensive review of current techniques, relevant datasets, and evaluation metrics across these five core areas. To clarify the gap between the proposed EgoProceAssist and existing VLM-based assistants, we conduct novel experiments to provide a comprehensive evaluation of representative VLM-based methods. Through these findings and our technical analysis, we discuss the challenges ahead and suggest future research directions. Furthermore, an exhaustive list of this study is publicly available in an active repository that continuously collects the latest work: https://github.com/z1oong/Building-Egocentric-Procedural-AI-Assistant.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
