Leveraging Foundation Model Automatic Data Augmentation Strategies and Skeletal Points for Hands Action Recognition in Industrial Assembly Lines
Faculty of Robot Science and Engineering, Northeastern University（东北大学机器人科学与工程学院）；Foshan Graduate School, Northeastern University（东北大学佛山研究生院）；College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）；The State Key Laboratory of Synthetical Automation for Process Industries, Northeastern University（东北大学流程工业综合自动化国家重点实验室）
On modern industrial assembly lines, many intelligent algorithms have been developed to replace or supervise workers. However, we found that there were bottlenecks in both training datasets and real-time performance when deploying algorithms on actual assembly line. Therefore, we developed a promising strategy for expanding industrial datasets, which utilized large models with strong generalization abilities to achieve efficient, high-quality, and large-scale dataset expansion, solving the problem of insufficient and low-quality industrial datasets. We also applied this strategy to video action recognition. We proposed a method of converting hand action recognition problems into hand skeletal trajectory classification problems, which solved the real-time performance problem of industrial algorithms. In the "hand movements during wire insertion" scenarios on the actual assembly line, the accuracy of hand action recognition reached 98.8\%. We conducted detailed experimental analysis to demonstrate the effectiveness and superiority of the method, and deployed the entire process on Midea's actual assembly line.
Bosch Rexroth AG（博世力士乐股份公司）；University of Applied Sciences Darmstadt（达姆施塔特应用科学大学）
This work presents the Industrial Hand Action Dataset V1, an industrial assembly dataset consisting of 12 classes with 459,180 images in the basic version and 2,295,900 images after spatial augmentation. Compared to other freely available datasets tested, it has an above-average duration and, in addition, meets the technical and legal requirements for industrial assembly lines. Furthermore, the dataset contains occlusions, hand-object interaction, and various fine-grained human hand actions for industrial assembly tasks that were not found in combination in examined datasets. The recorded grou…
Beijing A&E Technologies（北京A&E科技公司）；University of Pennsylvania（宾夕法尼亚大学）；Experian Data Lab（益博睿数据实验室）
Many annotation problems in computer vision can be phrased as integer linear programs (ILPs). The use of standard industrial solvers does not to exploit the underlying structure of such problems eg, the skeleton in pose estimation. The leveraging of the underlying structure in conjunction with industrial solvers promises increases in both speed and accuracy. Such structure can be exploited using Bender's decomposition, a technique from operations research, that solves complex ILPs or mixed integer linear programs by decomposing them into sub-problems that communicate via a master problem. The …
A vision-based framework for human behavior understanding in industrial assembly lines
Institute of Computer Science, Foundation for Research and Technology - Hellas（希腊研究与技术基金会计算机科学研究所）；School of Rural Surveying and Geoinformatics Engineering, National Technical University of Athens（雅典国立技术大学乡村测量与地理信息工程学院）；Institute of Communication and Computer Systems, National Technical University of Athens（雅典国立技术大学通信与计算机系统研究所）
AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects
Rutgers, The State University of New Jersey（新泽西罗格斯大学）；The Australian National University（澳大利亚国立大学）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔兰根-纽伦堡弗里德里希-亚历山大大学）；Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室）
Group-Mix SAM: Lightweight Solution for Industrial Assembly Line Applications
Northeastern University（东北大学）；Faculty of Robot Science and Engineering, Northeastern University（东北大学机器人科学与工程学院）；Foshan Graduate School, Northeastern University（东北大学佛山研究生院）；College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）；The State Key Laboratory of Synthetical Automation for Process Industries, Northeastern University（东北大学流程工业综合自动化国家重点实验室）
Highly constrained manipulation tasks continue to be challenging for autonomous robots as they require high levels of precision, typically less than 1mm, which is often incompatible with what can be achieved by traditional perception systems. This paper demonstrates that the combination of state-of-the-art object tracking with passively adaptive mechanical hardware can be leveraged to complete precision manipulation tasks with tight, industrially-relevant tolerances (0.25mm). The proposed control method closes the loop through vision by tracking the relative 6D pose of objects in the relevant …
The basic problem of shape complementarity analysis appears fundamental to applications as diverse as mechanical design, assembly automation, robot motion planning, micro- and nano-fabrication, protein-ligand binding, and rational drug design. However, the current challenge lies in the lack of a general mathematical formulation that applies to objects of arbitrary shape. We propose that a measure of shape complementarity can be obtained from the extent of approximate overlap between shape skeletons. A space-continuous implicit generalization of the skeleton, called the skeletal density functio…
Most of industrial robotic assembly tasks today require fixed initial conditions for successful assembly. These constraints induce high production costs and low adaptability to new tasks. In this work we aim towards flexible and adaptable robotic assembly by using 3D CAD models for all parts to be assembled. We focus on a generic assembly task - the Siemens Innovation Challenge - in which a robot needs to assemble a gear-like mechanism with high precision into an operating system. To obtain the millimeter-accuracy required for this task and industrial settings alike, we use a depth camera moun…
ProMQA-Assembly: Multimodal Procedural QA Dataset on Assembly
Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）；National Institute of Advanced Industrial Science and Technology (AIST)（国立研究开发法人产业技术综合研究所（AIST））
SmartFactory-KL（凯泽斯劳滕智能工厂）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
Detecting small objects in video streams of head-worn augmented reality devices in near real-time is a huge challenge: training data is typically scarce, the input video stream can be of limited quality, and small objects are notoriously hard to detect. In industrial scenarios, however, it is often possible to leverage contextual knowledge for the detection of small objects. Furthermore, CAD data of objects are typically available and can be used to generate synthetic training data. We describe a near real-time small object detection pipeline for egocentric perception in a manual assembly scen…
IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants
Fraunhofer IPK（弗劳恩霍夫研究所）；Technical University of Berlin（柏林技术大学）；University of Tübingen（图宾根大学）；RWTH Aachen University（亚琛工业大学）；Leibniz University Hannover（汉诺威莱布尼茨大学）
