Fiducial markers have been broadly used to identify objects or embed messages that can be detected by a camera. Primarily, existing detection methods assume that markers are printed on ideally planar surfaces. Markers often fail to be recognized due to various imaging artifacts of optical/perspective distortion and motion blur. To overcome these limitations, we propose a novel deformable fiducial marker system that consists of three main parts: First, a fiducial marker generator creates a set of free-form color patterns to encode significantly large-scale information in unique visual codes. Second, a differentiable image simulator creates a training dataset of photorealistic scene images with the deformed markers, being rendered during optimization in a differentiable manner. The rendered images include realistic shading with specular reflection, optical distortion, defocus and motion blur, color alteration, imaging noise, and shape deformation of markers. Lastly, a trained marker detector seeks the regions of interest and recognizes multiple marker patterns simultaneously via inverse deformation transformation. The deformable marker creator and detector networks are jointly optimized via the differentiable photorealistic renderer in an end-to-end manner, allowing us to robustly recognize a wide range of deformable markers with high accuracy. Our deformable marker system is capable of decoding 36-bit messages successfully at ~29 fps with severe shape deformation. Results validate that our system significantly outperforms the traditional and data-driven marker methods. Our learning-based marker system opens up new interesting applications of fiducial markers, including cost-effective motion capture of the human body, active 3D scanning using our fiducial markers' array as structured light patterns, and robust augmented reality rendering of virtual objects on dynamic surfaces.
Towards Markerless Intraoperative Tracking of Deformable Spine Tissue
The Hamlyn Centre for Robotic Surgery, Imperial College London, London, UK, Department of Computing, Imperial College London, London, UK Department of Mechanical Engineering, Imperial College London, London, UK Department of Surgery \& Cancer, Imperial College London, London, UK Department of Neurosurgery, IRCCS Humanitas Research Hospital, Rozzano, Lombardy, Italy Department of Biomedical Sciences, Humanitas University, Pieve Emanuele, Milan, Italy. Stanford University School of Medicine, Stanford University, California, USA
Existing fiducial markers solutions are designed for efficient detection and decoding, however, their ability to stand out in natural environments is difficult to infer from relatively limited analysis. Furthermore, worsening performance in challenging image capture scenarios - such as poor exposure, motion blur, and off-axis viewing - sheds light on their limitations. E2ETag introduces an end-to-end trainable method for designing fiducial markers and a complimentary detector. By introducing back-propagatable marker augmentation and superimposition into training, the method learns to generate …
Fiducial markers provide better-defined features than the ones naturally available in the scene. For this reason, they are widely utilized in computer vision applications where reliable pose estimation is required. Factors such as imaging noise and subtle changes in illumination induce jitter on the estimated pose. Jitter impairs robustness in vision and robotics applications, and deteriorates the sense of presence and immersion in AR/VR applications. In this paper, we propose STag, a fiducial marker system that provides stable pose estimation. STag is designed to be robust against jitter fact…
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Current fiducial marker detection algorithms rely on marker IDs for false positive rejection. Time is wasted on potential detections that will eventually be rejected as false positives. We introduce ChromaTag, a fiducial marker and detection algorithm designed to use opponent colors to limit and quickly reject initial false detections and grayscale for precise localization. Through experiments, we show that ChromaTag is significantly faster than current fiducial markers while achieving similar or better detection accuracy. We also show how tag size and viewing direction effect detection accura…
The fiducial marker system for LiDAR is crucial for the robotic application but it is still rare to date. In this paper, an Intensity Image-based LiDAR Fiducial Marker (IILFM) system is developed. This system only requires an unstructured point cloud with intensity as the input and it has no restriction on marker placement and shape. A marker detection method that locates the predefined 3D fiducials in the point cloud through the intensity image is introduced. Then, an approach that utilizes the detected 3D fiducials to estimate the LiDAR 6-DOF pose that describes the transmission from the wor…
Massachusetts Institute of Technology（麻省理工学院）；Microsoft（微软）
Adding fiducial markers to a scene is a well-known strategy for making visual localization algorithms more robust. Traditionally, these marker locations are selected by humans who are familiar with visual localization techniques. This paper explores the problem of automatic marker placement within a scene. Specifically, given a predetermined set of markers and a scene model, we compute optimized marker positions within the scene that can improve accuracy in visual localization. Our main contribution is a novel framework for modeling camera localizability that incorporates both natural scene fe…
A fiducial marker system usually consists of markers, a detection algorithm, and a coding system. The appearance of markers and the detection robustness are generally limited by the existing detection algorithms, which are hand-crafted with traditional low-level image processing techniques. Furthermore, a sophisticatedly designed coding system is required to overcome the shortcomings of both markers and detection algorithms. To improve the flexibility and robustness in various applications, we propose a general deep learning based framework, DeepTag, for fiducial marker design and detection. D…
We construct the first markerless deformable interaction dataset recording interactive motions of the hands and deformable objects, called HMDO (Hand Manipulation with Deformable Objects). With our built multi-view capture system, it captures the deformable interactions with multiple perspectives, various object shapes, and diverse interactive forms. Our motivation is the current lack of hand and deformable object interaction datasets, as 3D hand and deformable object reconstruction is challenging. Mainly due to mutual occlusion, the interaction area is difficult to observe, the visual feature…
Fiducial markers are commonly used in navigation assisted minimally invasive spine surgery (MISS) and they help transfer image coordinates into real world coordinates. In practice, these markers might be located outside the field-of-view (FOV), due to the limited detector sizes of C-arm cone-beam computed tomography (CBCT) systems used in intraoperative surgeries. As a consequence, reconstructed markers in CBCT volumes suffer from artifacts and have distorted shapes, which sets an obstacle for navigation. In this work, we propose two fiducial marker detection methods: direct detection from dis…
Visual fiducial systems are a key component of many robotics and AR/VR applications for 6-DOF monocular relative pose estimation and target identification. This paper presents LFTag, a visual fiducial system based on topological detection and relative position data encoding which optimizes data density within spatial frequency constraints. The marker is constructed to resolve rotational ambiguity, which combined with the robust geometric and topological false positive rejection, allows all marker bits to be used for data. When compared to existing state-of-the-art square binary markers (AprilT…
Fiducial markers have been playing an important role in augmented reality (AR), robot navigation, and general applications where the relative pose between a camera and an object is required. Here we introduce TopoTag, a robust and scalable topological fiducial marker system, which supports reliable and accurate pose estimation from a single image. TopoTag uses topological and geometrical information in marker detection to achieve higher robustness. Topological information is extensively used for 2D marker detection, and further corresponding geometrical information for ID decoding. Robust 3D p…
University of Pennsylvania School of Engineering and Applied Science（宾夕法尼亚大学工程与应用科学学院）
TagSLAM provides a convenient, flexible, and robust way of performing Simultaneous Localization and Mapping (SLAM) with AprilTag fiducial markers. By leveraging a few simple abstractions (bodies, tags, cameras), TagSLAM provides a front end to the GTSAM factor graph optimizer that makes it possible to rapidly design a range of experiments that are based on tags: full SLAM, extrinsic camera calibration with non-overlapping views, visual localization for ground truth, loop closure for odometry, pose estimation etc. We discuss in detail how TagSLAM initializes the factor graph in a robust way, an…
The Maimonides Biomedical Research Institute of Córdoba (IMIBIC)（科尔多瓦迈蒙尼德生物医学研究所）；University of Córdoba（科尔多瓦大学）；Université du Luxembourg（卢森堡大学）；Interdisciplinary Centre for Security, Reliability and Trust (SnT)（安全、可靠性与跨学科信任中心）
Event cameras are a new type of image sensors that output changes in light intensity (events) instead of absolute intensity values. They have a very high temporal resolution and a high dynamic range. In this paper, we propose a method to detect and decode binary square markers using an event camera. We detect the edges of the markers by detecting line segments in an image created from events in the current packet. The line segments are combined to form marker candidates. The bit value of marker cells is decoded using the events on their borders. To the best of our knowledge, no other approach …
Unveiling the Potential of iMarkers: Invisible Fiducial Markers for Advanced Robotics
Automation and Robotics Research Group, Interdisciplinary Centre for Security, Reliability, and Trust (SnT), University of Luxembourg（自动化与机器人研究组，安全、可靠性与信任跨学科中心（SnT），卢森堡大学）；Faculty of Science, Technology, and Medicine, University of Luxembourg（科学、技术与医学学院，卢森堡大学）；Department of Physics & Materials Science, University of Luxembourg（物理与材料科学系，卢森堡大学）；Institute for Advanced Studies, University of Luxembourg（先进研究学院，卢森堡大学）
Technical University of Vienna（维也纳技术大学）；AIT Austrian Institute of Technology（奥地利技术研究院）
Motion and dynamic environments, especially under challenging lighting conditions, are still an open issue for robust robotic applications. In this paper, we propose an end-to-end pipeline for real-time, low latency, 6 degrees-of-freedom pose estimation of fiducial markers. Instead of achieving a pose estimation through a conventional frame-based approach, we employ the high-speed abilities of event-based sensors to directly refine the spatial transformation, using consecutive events. Furthermore, we introduce a novel two-way verification process for detecting tracking errors by backtracking t…
Autonomous Robots Lab, University of Nevada, Reno（内华达大学雷诺分校自主机器人实验室）
For robotic inspection tasks in known environments fiducial markers provide a reliable and low-cost solution for robot localization. However, detection of such markers relies on the quality of RGB camera data, which degrades significantly in the presence of visual obscurants such as fog and smoke. The ability to navigate known environments in the presence of obscurants can be critical for inspection tasks especially, in the aftermath of a disaster. Addressing such a scenario, this work proposes a method for the design of fiducial markers to be used with thermal cameras for the pose estimation …
Image-based fiducial markers are useful in problems such as object tracking in cluttered or textureless environments, camera (and multi-sensor) calibration tasks, and vision-based simultaneous localization and mapping (SLAM). The state-of-the-art fiducial marker detection algorithms rely on the consistency of the ambient lighting. This paper introduces LiDARTag, a novel fiducial tag design and detection algorithm suitable for light detection and ranging (LiDAR) point clouds. The proposed method runs in real-time and can process data at 100 Hz, which is faster than the currently available LiDAR…
Fiducial systems provide a computationally cheap way for mobile robots to estimate the pose of objects, or their own pose, using just a monocular camera. However, the orientation component of the pose of fiducial markers is unreliable, which can have destructive effects in autonomous drone landing on landing pads marked with fiducial markers. This paper evaluates the April Tag and WhyCode fiducial systems in terms of orientation ambiguity and detection rate on embedded hardware. We test 2 April Tag variants - 1 default and 1 custom - and 3 Whycode variants - 1 default and 2 custom. We determin…
With camera resolution ever increasing and the need to rapidly recalibrate robotic platforms in less than ideal environments, there is a need for faster and more robust chessboard fiducial marker detectors. A new chessboard detector is proposed that is specifically designed for: high resolution images, focus/motion blur, harsh lighting conditions, and background clutter. This is accomplished using a new x-corner detector, where for the first time blur is estimated and used in a novel way to enhance corner localization, edge validation, and connectivity. Performance is measured and compared aga…
Instituto Maimónides de Investigación en Biomedicina (IMIBIC)（迈蒙尼德生物医学研究所）；Córdoba University（科尔多瓦大学）；Health Time；Clínica las Nieves（拉斯涅韦斯诊所）
BACKGROUND AND OBJECTIVE: Patient positioning is a crucial step in radiation therapy, for which non-invasive methods have been developed based on surface reconstruction using optical 3D imaging. However, most solutions need expensive specialized hardware and a careful calibration procedure that must be repeated over time.This paper proposes a fast and cheap patient positioning method based on inexpensive consumer level RGB-D sensors. METHODS: The proposed method relies on a 3D reconstruction approach that fuses, in real-time, artificial and natural visual landmarks recorded from a hand-held RG…
Monocular Marker-free Patient-to-Image Intraoperative Registration for Cochlear Implant Surgery
Dept. of Computer Science, Vanderbilt University（计算机科学系，范德比尔特大学）；Dept. of Electrical and Computer Engineering, Vanderbilt University（电气与计算机工程系，范德比尔特大学）
University of Utah（犹他大学）；University of Maryland, School of Medicine（马里兰大学医学院）
Radiation therapy presents a need for dynamic tracking of a target tumor volume. Fiducial markers such as implanted gold seeds have been used to gate radiation delivery but the markers are invasive and gating significantly increases treatment time. Pretreatment acquisition of a respiratory correlated 4DCT allows for determination of accurate motion tracking which is useful in treatment planning. We design a patient-specific motion subspace and a deep convolutional neural network to recover anatomical positions from a single fluoroscopic projection in real-time. We use this deep network to appr…
Centrum Wiskunde & Informatica（荷兰国家数学与计算机科学研究中心）；Leiden University（莱顿大学）；Thermo Fisher Scientific（赛默飞世尔科技）；Leiden Institute of Advanced Computer Science（莱顿高级计算机科学研究院）
Tilt-series alignment is crucial to obtaining high-resolution reconstructions in cryo-electron tomography. Beam-induced local deformation of the sample is hard to estimate from the low-contrast sample alone, and often requires fiducial gold bead markers. The state-of-the-art approach for deformation estimation uses (semi-)manually labelled marker locations in projection data to fit the parameters of a polynomial deformation model. Manually-labelled marker locations are difficult to obtain when data are noisy or markers overlap in projection data. We propose an alternative mathematical approach…
VERNIER: an open-source software pushing marker pose estimation down to the micrometer and nanometer scales
Université Marie et Louis Pasteur, SupMicroTech, CNRS, Institut FEMTO-ST（巴黎玛丽-路易·巴斯德大学、SupMicroTech、法国国家科学研究中心、FEMTO-ST研究所）；National Institute of Advanced Industrial Science and Technology, CNRS-AIST JRL, IRL, Tsukuba, Japan（日本国家先进工业科学与技术研究院、法国国家科学研究中心-AIST JRL、IRL、茨口）
Improving Deformable Image Registration Accuracy through a Hybrid Similarity Metric and CycleGAN Based Auto-Segmentation
Emory University（埃默里大学）；Winship Cancer Institute（温希普癌症研究所）；Drexel University（德雷塞尔大学）；Massachusetts General Hospital（马萨诸塞州总医院）；Harvard Medical School（哈佛医学院）
2D-3D Deformable Image Registration of Histology Slide and Micro-CT with ML-based Initialization
ImFusion GmbH（ImFusion有限公司）；Technical University of Munich（慕尼黑工业大学）；Institute of Pathology, Heinrich Heine University and University Hospital of Düsseldorf（海因里希·海涅大学与杜塞尔多夫大学医院病理研究所）；Fraunhofer Institute for High-Speed Dynamics, Ernst-Mach-Institut（弗劳恩霍夫高速动力学研究所（恩斯特·马赫研究所））
Johns Hopkins University（约翰斯·霍普金斯大学）；Ludwig-Maximilians-Universität München（慕尼黑大学）；Johns Hopkins Hospital（约翰斯·霍普金斯医院）；Technische Universität München（慕尼黑工业大学）
Fluoroscopic X-ray guidance is a cornerstone for percutaneous orthopaedic surgical procedures. However, two-dimensional observations of the three-dimensional anatomy suffer from the effects of projective simplification. Consequently, many X-ray images from various orientations need to be acquired for the surgeon to accurately assess the spatial relations between the patient's anatomy and the surgical tools. In this paper, we present an on-the-fly surgical support system that provides guidance using augmented reality and can be used in quasi-unprepared operating rooms. The proposed system build…
