Human comprehension of a video stream is naturally broad: in a few instants, we are able to understand what is happening, the relevance and relationship of objects, and forecast what will follow in the near future, everything all at once. We believe that - to effectively transfer such an holistic perception to intelligent machines - an important role is played by learning to correlate concepts and to abstract knowledge coming from different tasks, to synergistically exploit them when learning novel skills. To accomplish this, we seek for a unified approach to video understanding which combines shared temporal modelling of human actions with minimal overhead, to support multiple downstream tasks and enable cooperation when learning novel skills. We then propose EgoPack, a solution that creates a collection of task perspectives that can be carried across downstream tasks and used as a potential source of additional insights, as a backpack of skills that a robot can carry around and use when needed. We demonstrate the effectiveness and efficiency of our approach on four Ego4D benchmarks, outperforming current state-of-the-art methods.
EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understanding
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Peng Cheng Laboratory（鹏城实验室）；School of Artificial Intelligence, Sun Yat-sen University（中山大学人工智能学院）
mmEgoHand: Egocentric Hand Pose Estimation and Gesture Recognition with Head-mounted Millimeter-wave Radar and IMU
School of Software Engineering, Xi’an Jiaotong University（软件工程学院，西安交通大学）；School of Cyber Science and Engineering, Xi’an Jiaotong University（网络科学与工程学院，西安交通大学）；School of Computer Science and Technology, Xi’an Jiaotong University（计算机科学与技术学院，西安交通大学）；College of Computer Science and Technology, Zhejiang University（计算机科学与技术学院，浙江大学）
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
FAIR, Meta（Meta FAIR实验室）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加利福尼亚大学伯克利分校）；Project Aria, Meta（Meta Project Aria）；University of Bristol（布里斯托尔大学）；IIIT, Hyderabad（印度信息技术研究所海得拉巴分校）；University of Illinois, Urbana Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Minnesota（明尼苏达大学）；National University of Singapore（新加坡国立大学）；Universidad de los Andes（安第斯大学）；Simon Fraser University（西蒙菲莎大学）；University of North Carolina, Chapel Hill（北卡罗来纳大学教堂山分校）
EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；The University of Tokyo（东京大学）；Fudan University（复旦大学）；Nanjing University（南京大学）
METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Model
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机学院，北京大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World
OpenGVLab, Shanghai AI Laboratory（上海人工智能实验室OpenGVLab）；The University of Tokyo（东京大学）；Nanjing University（南京大学）；Fudan University（复旦大学）；Zhejiang University（浙江大学）；USTC（中国科学技术大学）；SIAT, CAS（中国科学院深圳先进技术研究院）
Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Qi Zhi Institute（上海期智研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Institute of Technology（北京理工大学）
