Autonomous assembly is a crucial capability for robots in many applications. For this task, several problems such as obstacle avoidance, motion planning, and actuator control have been extensively studied in robotics. However, when it comes to task specification, the space of possibilities remains underexplored. Towards this end, we introduce a novel problem, single-image-guided 3D part assembly, along with a learningbased solution. We study this problem in the setting of furniture assembly from a given complete set of parts and a single image depicting the entire assembled object. Multiple challenges exist in this setting, including handling ambiguity among parts (e.g., slats in a chair back and leg stretchers) and 3D pose prediction for parts and part subassemblies, whether visible or occluded. We address these issues by proposing a two-module pipeline that leverages strong 2D-3D correspondences and assembly-oriented graph message-passing to infer part relationships. In experiments with a PartNet-based synthetic benchmark, we demonstrate the effectiveness of our framework as compared with three baseline approaches.
In this work, we propose a framework called Auto-Assembly for automated robotic assembly from design files and demonstrate a practical implementation on modular parts joined by fastening using a robotic cell consisting of two robots. We show the flexibility of the approach by testing it on different input designs. Auto-Assembly consists of several parts: design analysis, assembly sequence generation, bill-of-process (BOP) generation, conversion of the BOP to control code, path planning, simulation, and execution of the control code to assemble parts in the physical environment.
Backward Layout Search for Sequence-Constrained Robotic Assembly
College of Computer and Information Sciences, Fujian Agriculture and Forestry University（福建农林大学计算机与信息学院）；School of Mechatronic Engineering and Automation, Shanghai University（上海大学机电工程与自动化学院）；Department of System Innovation, Graduate School of Engineering Science, Osaka University（大阪大学工程科学研究生院系统创新系）
Osaka University（大阪大学）；National Inst. of AIST（国立先进工业科学技术研究所）
Planning dual-arm assembly of more than three objects is a challenging Task and Motion Planning (TAMP) problem. The assembly planner shall consider not only the pose constraints of objects and robots, but also the gravitational constraints that may break the finished part. This paper proposes a planner to plan the dual-arm assembly of more than three objects. It automatically generates the grasp configurations and assembly poses, and simultaneously searches and backtracks the grasp space and assembly space to accelerate the motion planning of robot arms. Meanwhile, the proposed method consider…
Dhirubhai Ambani Institute of Information and Communication Technology（达鲁巴伊·安巴尼信息与通信技术学院）
Self-assembly is a process found everywhere in the Nature. In particular, it is known that DNA self-assembly is Turing universal. Thus one can do arbitrary computations or build nano-structures using DNA self-assembly. In order to understand the DNA self-assembly process, many mathematical models have been proposed in the literature. In particular, abstract Tile Assembly Model (aTAM) received much attention. In this work, we investigate pixel pattern generation using aTAM. For a given image, a tile assembly system is given which can generate the image by self-assembly process. We also consider…
Gwangju Institute of Science and Technology (GIST)（光州科学技术院）
Understanding assembly instruction has the potential to enhance the robot s task planning ability and enables advanced robotic applications. To recognize the key components from the 2D assembly instruction image, We mainly focus on segmenting the speech bubble area, which contains lots of information about instructions. For this, We applied Cascade Mask R-CNN and developed a context-aware data augmentation scheme for speech bubble segmentation, which randomly combines images cuts by considering the context of assembly instructions. We showed that the proposed augmentation scheme achieves a bet…
College of Computer Science, Dalian University of Technology（大连理工大学计算机学院）
Image matting is a long-standing problem in computer graphics and vision, mostly identified as the accurate estimation of the foreground in input images. We argue that the foreground objects can be represented by different-level information, including the central bodies, large-grained boundaries, refined details, etc. Based on this observation, in this paper, we propose a multi-scale information assembly framework (MSIA-matte) to pull out high-quality alpha mattes from single RGB images. Technically speaking, given an input image, we extract advanced semantics as our subject content and retain…
Most of industrial robotic assembly tasks today require fixed initial conditions for successful assembly. These constraints induce high production costs and low adaptability to new tasks. In this work we aim towards flexible and adaptable robotic assembly by using 3D CAD models for all parts to be assembled. We focus on a generic assembly task - the Siemens Innovation Challenge - in which a robot needs to assemble a gear-like mechanism with high precision into an operating system. To obtain the millimeter-accuracy required for this task and industrial settings alike, we use a depth camera moun…
Vision-based robotic assembly is a crucial yet challenging task as the interaction with multiple objects requires high levels of precision. In this paper, we propose an integrated 6D robotic system to perceive, grasp, manipulate and assemble blocks with tight tolerances. Aiming to provide an off-the-shelf RGB-only solution, our system is built upon a monocular 6D object pose estimation network trained solely with synthetic images leveraging physically-based rendering. Subsequently, pose-guided 6D transformation along with collision-free assembly is proposed to construct any designed structure …
A multi-layer image is more valuable than a single-layer image from a graphic designer's perspective. However, most of the proposed image generation methods so far focus on single-layer images. In this paper, we propose MontageGAN, which is a Generative Adversarial Networks (GAN) framework for generating multi-layer images. Our method utilized a two-step approach consisting of local GANs and global GAN. Each local GAN learns to generate a specific image layer, and the global GAN learns the placement of each generated image layer. Through our experiments, we show the ability of our method to ge…
Shishido&Associates；SK Intellectual Property Law Firm（SK知识产权律师事务所）；Enspirea LLC（Enspirea有限责任公司）
Deep learning has recently been applied to optical music recognition (OMR). However, currently OMR processing from various sheet music images still lacks precision to be widely applicable. Here, we present an MMdA (Measure-based Multimodal deep learning (DL)-driven Assembly) method allowing for end-to-end OMR processing from various images including inclined photo images. Using this method, measures are extracted by a deep learning model, aligned, and resized to be used for inference of given musical symbol components by using multiple deep learning models in sequence or in parallel. Use of ea…
AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents
The Australian National University（澳大利亚国立大学）；Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室（MERL））；Friedrich-Alexander-Universität Erlangen-Nürnberg（弗里德里希-亚历山大大学埃尔朗根-纽伦堡）；Tsinghua University（清华大学）
Bosch Rexroth AG（博世力士乐股份公司）；University of Applied Sciences Darmstadt（达姆施塔特应用科学大学）
This work presents the Industrial Hand Action Dataset V1, an industrial assembly dataset consisting of 12 classes with 459,180 images in the basic version and 2,295,900 images after spatial augmentation. Compared to other freely available datasets tested, it has an above-average duration and, in addition, meets the technical and legal requirements for industrial assembly lines. Furthermore, the dataset contains occlusions, hand-object interaction, and various fine-grained human hand actions for industrial assembly tasks that were not found in combination in examined datasets. The recorded grou…
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；The Chinese University of Hong Kong（香港中文大学）
In industry assembly lines, parts feeding machines are widely employed as the prologue of the whole procedure. They play the role of sorting the parts randomly placed in bins to the state with specified pose. With the help of the parts feeding machines, the subsequent assembly processes by robot arm can always start from the same condition. Thus it is expected that function of parting feeding machine and the robotic assembly can be integrated with one robot arm. This scheme can provide great flexibility and can also contribute to reduce the cost. The difficulties involved in this scheme lie in…
Visual segmentation has seen tremendous advancement recently with ready solutions for a wide variety of scene types, including human hands and other body parts. However, focus on segmentation of human hands while performing complex tasks, such as manual assembly, is still severely lacking. Segmenting hands from tools, work pieces, background and other body parts is extremely difficult because of self-occlusions and intricate hand grips and poses. In this paper we introduce BusyHands, a large open dataset of pixel-level annotated images of hands performing 13 different tool-based assembly tasks…
Boston College（波士顿学院）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
Today's person detection methods work best when people are in common upright poses and appear reasonably well spaced out in the image. However, in many real images, that's not what people do. People often appear quite close to each other, e.g., with limbs linked or heads touching, and their poses are often not pedestrian-like. We propose an approach to detangle people in multi-person images. We formulate the task as a region assembly problem. Starting from a large set of overlapping regions from body part semantic segmentation and generic object proposals, our optimization approach reassembles…
Continuous Expert Assembly: Instance-Conditioned Low-Rank Residuals for All-in-One Image Restoration
Southern University of Science and Technology（南方科技大学）；Shenzhen University（深圳大学）；Shenzhen University of Advanced Technology（深圳大学先进技术学院）；Xi’an Jiaotong University（西安交通大学）
Tongji University（同济大学）；ByteDance AI Lab（字节跳动人工智能实验室）
It is desirable to enable robots capable of automatic assembly. Structural understanding of object parts plays a crucial role in this task yet remains relatively unexplored. In this paper, we focus on the setting of furniture assembly from a complete set of part geometries, which is essentially a 6-DoF part pose estimation problem. We propose a multi-layer transformer-based framework that involves geometric and relational reasoning between parts to update the part poses iteratively. We carefully design a unique instance encoding to solve the ambiguity between geometrically-similar parts so tha…
University of Waterloo（滑铁卢大学）；Waterloo Institute for Nanotechnology（滑铁卢纳米技术研究所）
A method for quantitative analysis of local pattern strength and defects in surface self-assembly imaging is presented and applied to images of stripe and hexagonal ordered domains. The presented method uses "shapelet" functions which were originally developed for quantitative analysis of images of galaxies ($\propto 10^{20}\mathrm{m}$). In this work, they are used instead to quantify the presence of translational order in surface self-assembled films ($\propto 10^{-9}\mathrm{m}$) through reformulation into "steerable" filters. The resulting method is both computationally efficient (with respe…
Human-designed visual manuals are crucial components in shape assembly activities. They provide step-by-step guidance on how we should move and connect different parts in a convenient and physically-realizable way. While there has been an ongoing effort in building agents that perform assembly tasks, the information in human-design manuals has been largely overlooked. We identify that this is due to 1) a lack of realistic 3D assembly objects that have paired manuals and 2) the difficulty of extracting structured information from purely image-based manuals. Motivated by this observation, we pre…
Shandong University（山东大学）；AICFVE, Beijing Film Academy（北京电影学院未来影像高精尖创新中心）；CFCS, Peking University（北京大学前沿计算研究中心）
Co-part segmentation is an important problem in computer vision for its rich applications. We propose an unsupervised learning approach for co-part segmentation from images. For the training stage, we leverage motion information embedded in videos and explicitly extract latent representations to segment meaningful object parts. More importantly, we introduce a dual procedure of part-assembly to form a closed loop with part-segmentation, enabling an effective self-supervision. We demonstrate the effectiveness of our approach with a host of extensive experiments, ranging from human bodies, hands…
Microsoft Research Asia（微软亚洲研究院）；Carnegie Mellon University（卡内基梅隆大学）
Instance segmentation is a challenging task aiming at classifying and segmenting all object instances of specific classes. While two-stage box-based methods achieve top performances in the image domain, they cannot easily extend their superiority into the video domain. This is because they usually deal with features or images cropped from the detected bounding boxes without alignment, failing to capture pixel-level temporal consistency. We embrace the observation that bottom-up methods dealing with box-free features could offer accurate spacial correlations across frames, which can be fully ut…
Multi-view images are acquired by a lensless compressive imaging architecture, which consists of an aperture assembly and multiple sensors. The aperture assembly consists of a two dimensional array of aperture elements whose transmittance can be individually controlled to implement a compressive sensing matrix. For each transmittance pattern of the aperture assembly, each of the sensors takes a measurement. The measurement vectors from the multiple sensors represent multi-view images of the same scene. We present theoretical framework for multi-view reconstruction and experimental results for …
In this paper, we propose a lensless compressive sensing imaging architecture. The architecture consists of two components, an aperture assembly and a sensor. No lens is used. The aperture assembly consists of a two dimensional array of aperture elements. The transmittance of each aperture element is independently controllable. The sensor is a single detection element, such as a single photo-conductive cell. Each aperture element together with the sensor defines a cone of a bundle of rays, and the cones of the aperture assembly define the pixels of an image. Each pixel value of an image is the…
Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention
Xi’an Jiaotong University（西安交通大学）；National Engineering Laboratory for Big Data Analytics（国家大数据分析工程实验室）；Nanyang Technological University（南洋理工大学）；Lenovo Research（联想研究院）；University of Massachusetts Boston（马萨诸塞大学波士顿分校）；Zhejiang University of Technology（浙江工业大学）
AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects
Rutgers, The State University of New Jersey（新泽西罗格斯大学）；The Australian National University（澳大利亚国立大学）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔兰根-纽伦堡弗里德里希-亚历山大大学）；Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室）
University of Nottingham（诺丁汉大学）；UPENN（宾夕法尼亚大学）；NIH（美国国立卫生研究院）
This paper investigates, using prior shape models and the concept of ball scale (b-scale), ways of automatically recognizing objects in 3D images without performing elaborate searches or optimization. That is, the goal is to place the model in a single shot close to the right pose (position, orientation, and scale) in a given image so that the model boundaries fall in the close vicinity of object boundaries in the image. This is achieved via the following set of key ideas: (a) A semi-automatic way of constructing a multi-object shape model assembly. (b) A novel strategy of encoding, via b-scal…
Stanford University（斯坦福大学）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；NVIDIA（英伟达）；Tsinghua University（清华大学）；Peking University（北京大学）；National University of Singapore（新加坡国立大学）
Shape assembly composes complex shapes geometries by arranging simple part geometries and has wide applications in autonomous robotic assembly and CAD modeling. Existing works focus on geometry reasoning and neglect the actual physical assembly process of matching and fitting joints, which are the contact surfaces connecting different parts. In this paper, we consider contacting joints for the task of multi-part assembly. A successful joint-optimized assembly needs to satisfy the bilateral objectives of shape structure and joint alignment. We propose a hierarchical graph learning approach comp…
In this paper, we propose a lensless compressive imaging architecture. The architecture consists of two components, an aperture assembly and a sensor. No lens is used. The aperture assembly consists of a two dimensional array of aperture elements. The transmittance of each aperture element is independently controllable. The sensor is a single detection element. A compressive sensing matrix is implemented by adjusting the transmittance of the individual aperture elements according to the values of the sensing matrix. The proposed architecture is simple and reliable because no lens is used. The …
A-SDM: Accelerating Stable Diffusion through Model Assembly and Feature Inheritance Strategies
Tsinghua University（清华大学）；Nankai University（南开大学）；Kuaishou Technology（快手科技）；Tianjin University of Finance and Economics（天津财经大学）；Imperial College London（伦敦帝国学院）
Autonomous part assembly is a challenging yet crucial task in 3D computer vision and robotics. Analogous to buying an IKEA furniture, given a set of 3D parts that can assemble a single shape, an intelligent agent needs to perceive the 3D part geometry, reason to propose pose estimations for the input parts, and finally call robotic planning and control routines for actuation. In this paper, we focus on the pose estimation subproblem from the vision side involving geometric and relational reasoning over the input part geometry. Essentially, the task of generative 3D part assembly is to predict …
ISC-Perception: A Hybrid Computer Vision Dataset for Object Detection in Novel Steel Assembly
School of Natural and Built Environment, Queen’s University Belfast（女王大学贝尔法斯特自然与建筑环境学院）；School of Civil & Environmental Engineering, and Construction Management, University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校土木与环境工程学院及建设管理学院）；School of Electronics, Electrical Engineering, and Computer Science, Queen’s University Belfast（女王大学贝尔法斯特电子、电气工程及计算机科学学院）；Department of Civil and Urban Engineering, New York University（纽约大学土木与城市工程系）
