发表机构
The University of Hong Kong; Tongji University（香港大学; 同济大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出NextMe-800数据集和NextAct基准，用于从数月自我中心视频中预测个人行为，通过开放词汇K步序列预测方法，评估不同模型在多个抽象层级上的预测能力。
AI 中文摘要
我们常常雄心勃勃地制定计划，却习惯性地行动，事后回想，如果早知道实际会做什么，我们是否会做出不同的计划。事后回顾为过去的决策提供了宝贵的视角，尽管我们常常希望在做出选择的那一刻就能模拟这种事后判断。如果系统能从个人历史中生成合理的轨迹，这些预览或许能帮助人们制定更现实的计划并做出更明智的决策。我们推出了NextMe-800，这是一个约800小时的自我中心数据集，来自一位志愿者在126天内的记录，包含1赫兹图像、注视和音频，并以从原子动作到主要活动的五个层级抽象水平进行标注。我们将个性化行为预测定义为开放词汇的K步序列预测，并构建了NextAct，一个包含1500个测试点的基准，结合了NextMe-800与多人EgoLife数据集。使用基于嵌入的软编辑距离作为评估指标，我们评估了不同模型在不同抽象水平和预测视界上预测个人行为的能力。NextMe-800和NextAct提供了一个跨越数月的数据资源和评估框架，用于研究从自我中心观察中能预测个人行为的程度。
英文摘要
We often plan ambitiously yet act habitually and wonder, in retrospect, whether we would have planned differently had we known what we would actually do. Hindsight offers a valuable perspective on past decisions, although we often wish we could have simulated hindsight at the moment of choosing. If a system could generate plausible trajectories from one's personal history, such previews might help people formulate more realistic plans and make better informed decisions. We introduce NextMe-800, an approximately 800-hour first-person dataset from one volunteer over 126 days with 1 Hz images, gaze, and audio, captioned at five hierarchical abstraction levels from atomic actions to major activities. We formulate personalized action anticipation as open-vocabulary K-step sequence prediction and construct NextAct, a 1,500-point benchmark combining NextMe-800 with the multi-person EgoLife dataset. Using an embedding-based soft edit distance as the metric, we evaluate how well different models can anticipate personal behavior across abstraction levels and prediction horizons. NextMe-800 and NextAct provide a months-long resource and evaluation framework for studying how far ahead personal behavior can be anticipated from egocentric observation.
Comments24 pages, 7 figures. Dataset and benchmark: https://huggingface.co/datasets/mmm8383/NextMe-800 ; project page: https://kkkkawayi.github.io/nextme-800/
