In this paper we address the task of recognizing assembly actions as a structure (e.g. a piece of furniture or a toy block tower) is built up from a set of primitive objects. Recognizing the full range of assembly actions requires perception at a level of spatial detail that has not been attempted in the action recognition literature to date. We extend the fine-grained activity recognition setting to address the task of assembly action recognition in its full generality by unifying assembly actions and kinematic structures within a single framework. We use this framework to develop a general method for recognizing assembly actions from observation sequences, along with observation features that take advantage of a spatial assembly's special structure. Finally, we evaluate our method empirically on two application-driven data sources: (1) An IKEA furniture-assembly dataset, and (2) A block-building dataset. On the first, our system recognizes assembly actions with an average framewise accuracy of 70% and an average normalized edit distance of 10%. On the second, which requires fine-grained geometric reasoning to distinguish between assemblies, our system attains an average normalized edit distance of 23% -- a relative improvement of 69% over prior work.
Vrije University of Amsterdam University of Amsterdam Centrum Wiskunde \& Informatic;University of Amsterdam National Institute of Informatics Centrum Wiskunde \& Informatic;Centrum Wiskunde \& Informatic Leiden University University College London;Vrije University of Amsterdam;Centrum Wiskunde \& Informatica Technische Universiteit Delft Guangzhou University;Vrije University of Amsterdam Centrum Wiskunde \& Informatic
Weakly Supervised Temporal Convolutional Networks for Fine-grained Surgical Activity Recognition
University of Verona(维罗纳大学);University of Strasbourg(斯特拉斯堡大学);CNRS(法国国家科学研究中心);IHU Strasbourg(斯特拉斯堡大学医院研究所);University Hospital of Strasbourg(斯特拉斯堡大学医院);IRCAD(斯特拉斯堡微创外科研究中心);Fondazione Policlinico Universitario Agostino Gemelli IRCCS(阿戈斯蒂诺·杰梅利大学医院基金会IRCCS)
Recognizing Fine-Grained and Composite Activities using Hand-Centric Features and Script Data
UC Berkeley(加州大学伯克利分校);ICSI(国际计算机科学研究所);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所);Saarland University(萨尔大学);Technische Universität München(慕尼黑工业大学);Stanford University(斯坦福大学);SPIEGEL-Verlag(明镜出版社)
Probing Fine-Grained Action Understanding and Cross-View Generalization of Foundation Models
Institute for Artificial Intelligence, University of Stuttgart(斯图加特大学人工智能研究所);Institute for Anthropomatics and Robotics (IAR), KIT(卡尔斯鲁厄理工学院人形机器人与智能系统研究所)
EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching
University of Science and Technology of China(中国科学技术大学);Southeast University(东南大学);Alibaba Group(阿里巴巴集团);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);CAS Center for Excellence in Brain Science and Intelligence Technology(中国科学院脑科学与智能技术卓越创新中心)