TI-PREGO: Chain of Thought and In-Context Learning for Online Mistake Detection in PRocedural EGOcentric Videos
Sapienza University of Rome（萨皮恩扎罗马大学）；ItalAI；University of Catania（卡塔尼亚大学）；National Institute of Nuclear Physics - LNF（国家核物理研究所 - LNF）
Leonardo Plini, Luca Scofano, Edoardo De Matteis, Guido Maria D'Amely di Melendugno, Alessandro Flaborea, Andrea Sanchietti, Giovanni Maria Farinella, Fabio Gal… 展开作者
Leonardo Plini, Luca Scofano, Edoardo De Matteis, Guido Maria D'Amely di Melendugno, Alessandro Flaborea, Andrea Sanchietti, Giovanni Maria Farinella, Fabio Galasso, Antonino Furnari
Identifying procedural errors online from egocentric videos is a critical yet challenging task across various domains, including manufacturing, healthcare, and skill-based training. The nature of such mistakes is inherently open-set, as unforeseen or novel errors may occur, necessitating robust detection systems that do not rely on prior examples of failure. Currently, however, no technique effectively detects open-set procedural mistakes online.
We propose a dual branch architecture to address this problem in an online fashion: one branch continuously performs step recognition from the input egocentric video, while the other anticipates future steps based on the recognition module's output. Mistakes are detected as mismatches between the currently recognized action and the action predicted by the anticipation module. The recognition branch takes input frames, predicts the current action, and aggregates frame-level results into action tokens. The anticipation branch, specifically, leverages the solid pattern-matching capabilities of Large Language Models (LLMs) to predict action tokens based on previously predicted ones.
Given the online nature of the task, we also thoroughly benchmark the difficulties associated with per-frame evaluations, particularly the need for accurate and timely predictions in dynamic online scenarios.
Extensive experiments on two procedural datasets demonstrate the challenges and opportunities of leveraging a dual-branch architecture for mistake detection, showcasing the effectiveness of our proposed approach. In a thorough evaluation including recognition and anticipation variants and state-of-the-art models, our method reveals its robustness and effectiveness in online applications.
Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges
Department of EEE（电子工程系）；The Hong Kong Polytechnic University（香港理工大学）；RayNeo；Tsinghua University（清华大学）；Huazhong University of Science and Technology（华中科技大学）；Nanyang Technological University（南洋理工大学）
Exo2EgoDVC: Dense Video Captioning of Egocentric Procedural Activities Using Web Instructional Videos
The University of Tokyo（东京大学）；OMRON SINIC X Corp.（欧姆龙SINIC X公司）；National Institute of Advanced Industrial Science and Technology (AIST)（日本产业技术综合研究所）；LY Corporation（LY公司）
EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World
OpenGVLab, Shanghai AI Laboratory（上海人工智能实验室OpenGVLab）；The University of Tokyo（东京大学）；Nanjing University（南京大学）；Fudan University（复旦大学）；Zhejiang University（浙江大学）；USTC（中国科学技术大学）；SIAT, CAS（中国科学院深圳先进技术研究院）
Meta Reality Labs Research（Meta现实实验室）；National University of Singapore（新加坡国立大学）
Assembly101 is a new procedural activity dataset featuring 4321 videos of people assembling and disassembling 101 "take-apart" toy vehicles. Participants work without fixed instructions, and the sequences feature rich and natural variations in action ordering, mistakes, and corrections. Assembly101 is the first multi-view action dataset, with simultaneous static (8) and egocentric (4) recordings. Sequences are annotated with more than 100K coarse and 1M fine-grained action segments, and 18M 3D hand poses. We benchmark on three action understanding tasks: recognition, anticipation and temporal …
Ego-EXTRA: video-language Egocentric Dataset for EXpert-TRAinee assistance
Department of Mathematics and Computer Science - University of Catania（数学与计算机科学系 - 卡塔尼亚大学）；Next Vision s.r.l. - Spinoff of the University of Catania（Next Vision公司 - 卡塔尼亚大学衍生机构）；Meta Reality Labs Research（Meta现实实验室）
Columbia University（哥伦比亚大学）；Facebook AI Research（脸书人工智能研究院）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）；Dartmouth（达特茅斯学院）
In this paper we consider the problem of classifying fine-grained, multi-step activities (e.g., cooking different recipes, making disparate home improvements, creating various forms of arts and crafts) from long videos spanning up to several minutes. Accurately categorizing these activities requires not only recognizing the individual steps that compose the task but also capturing their temporal dependencies. This problem is dramatically different from traditional action classification, where models are typically optimized on videos that span only a few seconds and that are manually trimmed to…
IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants
Fraunhofer IPK（弗劳恩霍夫研究所）；Technical University of Berlin（柏林技术大学）；University of Tübingen（图宾根大学）；RWTH Aachen University（亚琛工业大学）；Leibniz University Hannover（汉诺威莱布尼茨大学）
How You Move Tells What You'll Do: Trajectory-Conditioned Egocentric Prediction
Khoury College of Computer Sciences, Northeastern University, Boston（北德文斯克学院，东北大学，波士顿）；Korea Advanced Institute of Science and Technology, Daejeon（韩国科学技术院，大田）；Department of Mathematics and Computer Science, University of Catania, Italy（卡塔尼亚大学数学与计算机科学系，意大利）
