Detecting small objects in video streams of head-worn augmented reality devices in near real-time is a huge challenge: training data is typically scarce, the input video stream can be of limited quality, and small objects are notoriously hard to detect. In industrial scenarios, however, it is often possible to leverage contextual knowledge for the detection of small objects. Furthermore, CAD data of objects are typically available and can be used to generate synthetic training data. We describe a near real-time small object detection pipeline for egocentric perception in a manual assembly scenario: We generate a training data set based on CAD data and realistic backgrounds in Unity. We then train a YOLOv4 model for a two-stage detection process: First, the context is recognized, then the small object of interest is detected. We evaluate our pipeline on the augmented reality device Microsoft Hololens 2.
Human-designed visual manuals are crucial components in shape assembly activities. They provide step-by-step guidance on how we should move and connect different parts in a convenient and physically-realizable way. While there has been an ongoing effort in building agents that perform assembly tasks, the information in human-design manuals has been largely overlooked. We identify that this is due to 1) a lack of realistic 3D assembly objects that have paired manuals and 2) the difficulty of extracting structured information from purely image-based manuals. Motivated by this observation, we pre…
ISC-Perception: A Hybrid Computer Vision Dataset for Object Detection in Novel Steel Assembly
School of Natural and Built Environment, Queen’s University Belfast（女王大学贝尔法斯特自然与建筑环境学院）；School of Civil & Environmental Engineering, and Construction Management, University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校土木与环境工程学院及建设管理学院）；School of Electronics, Electrical Engineering, and Computer Science, Queen’s University Belfast（女王大学贝尔法斯特电子、电气工程及计算机科学学院）；Department of Civil and Urban Engineering, New York University（纽约大学土木与城市工程系）
Visual segmentation has seen tremendous advancement recently with ready solutions for a wide variety of scene types, including human hands and other body parts. However, focus on segmentation of human hands while performing complex tasks, such as manual assembly, is still severely lacking. Segmenting hands from tools, work pieces, background and other body parts is extremely difficult because of self-occlusions and intricate hand grips and poses. In this paper we introduce BusyHands, a large open dataset of pixel-level annotated images of hands performing 13 different tool-based assembly tasks…
Visual defect detection is critical to ensure the quality of most products. However, the majority of small and medium-sized manufacturing enterprises still rely on tedious and error-prone human manual inspection. The main reasons include: 1) the existing automated visual defect detection systems require altering production assembly lines, which is time consuming and expensive 2) the existing systems require manually collecting defective samples and labeling them for a comparison-based algorithm or training a machine learning model. This introduces a heavy burden for small and medium-sized manu…
Brigham Young University（杨百翰大学）；Adobe Research（Adobe 研究院）
Hand segmentation and detection in truly unconstrained RGB-based settings is important for many applications. However, existing datasets are far from sufficient in terms of size and variety due to the infeasibility of manual annotation of large amounts of segmentation and detection data. As a result, current methods are limited by many underlying assumptions such as constrained environment, consistent skin color and lighting. In this work, we present Ego2Hands, a large-scale RGB-based egocentric hand segmentation/detection dataset that is semi-automatically annotated and a color-invariant comp…
IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants
Fraunhofer IPK（弗劳恩霍夫研究所）；Technical University of Berlin（柏林技术大学）；University of Tübingen（图宾根大学）；RWTH Aachen University（亚琛工业大学）；Leibniz University Hannover（汉诺威莱布尼茨大学）
Northwestern University（西北大学）；IBM T. J. Watson（IBM托马斯·J·沃森研究中心）
The way people look in terms of facial attributes (ethnicity, hair color, facial hair, etc.) and the clothes or accessories they wear (sunglasses, hat, hoodies, etc.) is highly dependent on geo-location and weather condition, respectively. This work explores, for the first time, the use of this contextual information, as people with wearable cameras walk across different neighborhoods of a city, in order to learn a rich feature representation for facial attribute classification, without the costly manual annotation required by previous methods. By tracking the faces of casual walkers on more t…
AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects
Rutgers, The State University of New Jersey（新泽西罗格斯大学）；The Australian National University（澳大利亚国立大学）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔兰根-纽伦堡弗里德里希-亚历山大大学）；Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室）
Manual annotations of temporal bounds for object interactions (i.e. start and end times) are typical training input to recognition, localization and detection algorithms. For three publicly available egocentric datasets, we uncover inconsistencies in ground truth temporal bounds within and across annotators and datasets. We systematically assess the robustness of state-of-the-art approaches to changes in labeled temporal bounds, for object interaction recognition. As boundaries are trespassed, a drop of up to 10% is observed for both Improved Dense Trajectories and Two-Stream Convolutional Neu…
In object recognition research, many commonly used datasets (e.g., ImageNet and similar) contain relatively sparse distributions of object instances and views, e.g., one might see a thousand different pictures of a thousand different giraffes, mostly taken from a few conventionally photographed angles. These distributional properties constrain the types of computational experiments that are able to be conducted with such datasets, and also do not reflect naturalistic patterns of embodied visual experience. As a contribution to the small (but growing) number of multi-view object datasets that h…
VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI
Tsinghua University（清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；Tencent Robotics X（腾讯Robotics X实验室）；University of Toronto（多伦多大学）；Zhili College, Tsinghua University（清华大学志道书院）
Stanford University（斯坦福大学）；Massachusetts Institute of Technology（麻省理工学院）；Google Research（谷歌研究院）
We study the problem of translating an image-based, step-by-step assembly manual created by human designers into machine-interpretable instructions. We formulate this problem as a sequential prediction task: at each step, our model reads the manual, locates the components to be added to the current shape, and infers their 3D poses. This task poses the challenge of establishing a 2D-3D correspondence between the manual image and the real 3D object, and 3D pose estimation for unseen 3D objects, since a new component to be added in a step can be an object built from previous steps. To address the…
PAWS: Perception of Articulation in the Wild at Scale from Egocentric Videos
Aalto University（阿尔托大学）；INSAIT, Sofia University（索菲亚大学INSAIT研究所）；ETH Zurich（苏黎世联邦理工学院）；TU Munich（慕尼黑工业大学）；MCML（慕尼黑机器学习中心）；ELLIS Institute Finland（芬兰ELLIS研究所）；University of Oulu（奥卢大学）
GazeBehavior Annotation Toolkit (GBAT): AI-powered toolkit for automatic annotation of egocentric eye-tracking and video data of child-caregiver interaction
Department of Psychology, University of Miami（迈阿密大学心理学系）；Northeastern University（东北大学）；Ecole Normale Supérieure, PSL University, EHESS, CNRS（巴黎高等师范学院（PSL大学）、EHESS、CNRS）；International Research Center for Neurointelligence (WPI-IRCN), The University of Tokyo Institutes for Advanced Study（神经智能国际研究中心（WPI-IRCN）、东京大学高级研究机构）
Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（人工智能教育部重点实验室，上海交通大学AI研究院）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（宁波数字孪生研究所，东部技术研究所）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative（宁波空间智能与数字衍生关键实验室）；MoE Key Lab of AI, School of Computer Science, Shanghai Jiao Tong University（人工智能教育部重点实验室，上海交通大学计算机学院）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Lenovo（联想公司）
For embodied agents, navigation is an important ability but not an isolated goal. Agents are also expected to perform specific tasks after reaching the target location, such as picking up objects and assembling them into a particular arrangement. We combine Vision-and-Language Navigation, assembling of collected objects, and object referring expression comprehension, to create a novel joint navigation-and-assembly task, named ArraMon. During this task, the agent (similar to a PokeMON GO player) is asked to find and collect different target objects one-by-one by navigating based on natural lang…
Aligning Step-by-Step Instructional Diagrams to Video Demonstrations
The Australian National University（澳大利亚国立大学）；Mitsubishi Electric Research Labs（三菱电机研究院）；Technion Israel Institute of Technology（以色列理工学院）；The Australian Institute for Machine Learning（澳大利亚机器学习研究所）
ESA: Energy-Based Shot Assembly Optimization for Automatic Video Editing
Sobey Media Intelligence Laboratory（索贝媒体智能实验室）；University of Electronic Science and Technology of China（电子科学与技术大学）；SiChuan University（四川大学）；Qinghai Normal University（青海师范大学）
